Display device, control method for display device, and computer program

ABSTRACT

An HMD includes an image display section configured to display an image to be visually recognizable through an outside scene. The HMD includes a position detecting section configured to recognize an input and a control section configured to cause the image display section to display information and change the display in the image display section according to the input recognized by the position detecting section.

This is a Continuation of application Ser. No. 15/130,281 filed Apr. 15,2016, which claims the benefit of Japanese Patent Application Nos.2015-135738, filed Jul. 7, 2015 and 2016-013249, filed Jan. 27, 2016,entire contents of which are incorporated by reference herein in itsentirety.

BACKGROUND 1. Technical Field

The invention relates to a display device, a control method for thedisplay device, and a computer program.

2. Related Art

There has been known a method of supporting execution of a workprocedure according to a scenario including information concerning thework procedure (see, for example, JP-A-2013-29985 (Patent Literature1)). In the method described in Patent Literature 1, a work list or thelike is displayed according to a scenario including an action sequenceconcerning details of the work procedure to support the execution of thework procedure. An execution state is displayed on a screen.

In the method described in Patent Literature 1, an operator performswork viewing the screen on which the work list or the like is displayed.In this case, the operator needs to view both of a place or a targetobject, which is a target of the work, and the screen. When informationrelated to a motion of a person who performs the motion targeting theplace or the object is shown to the person, the person (e.g., anoperator) has to, for example, move a visual line in order to view theplace or the object, which is the target of the motion, and theinformation related to the motion. Therefore, there is a demand for amethod capable of easily showing, when the person performs the motiontargeting the place or the object, the information concerning the motionto the person.

SUMMARY

An advantage of some aspects of the invention is to easily showinformation concerning a motion to a person who performs the motiontargeting a place or an object.

A display device according to an aspect of the invention includes: adisplay section configured to display an image to be visuallyrecognizable through an outside scene; an input section configured torecognize an input; and a control section configured to cause thedisplay section to display information and change the display in thedisplay section according to the input recognized by the input section.

According to the aspect, a person viewing the display device can changethe display by viewing the displayed information and performing an inputrecognizable by the input section. Consequently, a person who performs amotion targeting a place or an object can visually recognize the placeor the object, which is the target of the motion, through the displaysection and view, for example, information concerning the motion in thisstate. In this way, the person can view the target present in a realspace, which is an outside scene, and the information together even ifthe person does not perform a movement with a large load such as a largemovement of a visual line. Since the display is changed according to theinput, it is possible to sequentially provide new information by, forexample, changing the display according to the progress of the motion.

In the display device according to the aspect, the control section maycause the display section to display the item as the information on thebasis of item data including an item.

According to the aspect with this configuration, it is possible todisplay various numbers and various types of items.

In the display device according to the aspect, the item data may includea plurality of the items corresponding to motions, and, in therespective items, order of the item corresponding to execution order ofthe motions may be set.

According to the aspect with this configuration, it is possible toinform, with a display form of the item, the person viewing the displaydevice of the execution order of the motion.

In the display device according to the aspect, the control section mayarrange the plurality of items in a list format and cause the displaysection to display the plurality of items, and the item may be arrangedaccording to the order set for the item.

According to the aspect with this configuration, it is possible toinform, with the arrangement of the displayed plurality of items, theperson viewing the display device of the execution order of the motion.

In the display device according to the aspect, the control section maycause the display section to sequentially display the item in the orderset for the item.

According to the aspect with this configuration, it is possible toinform, with the display order of the item, the person viewing thedisplay device of the execution order of the motion.

In the display device according to the aspect, the control section maydisplay in association with the item, a checkbox indicating completionof the motion corresponding to the item and, after sequentiallydisplaying the item in the order set for the item, when detecting thecompletion of the notion corresponding to the item, change a displaystate of the checkbox to a state indicating the motion completion.

According to the aspect with this configuration, when informing theperson of the execution order of the motion with the display order ofthe item, it is possible to clearly inform the person of the completedmotion and clearly show a progress situation of the motion.

In the display device according to the aspect, the control section maycause the display section to display an image on the basis of order dataincluding information concerning the execution order of the motioncorresponding to the item included in the item data.

According to the aspect with this configuration, it is possible toinform, with the image, the person viewing the display device of theexecution order of the motion.

In the display device according to the aspect, the display device mayfurther include an order-data storing section configured to store theitem data and the order data in association with each other.

According to the aspect with this configuration, it is possible toeasily read out and use the data concerning the item concerning themotion and the data concerning the execution order of the motion.

In the display device according to the aspect, the control section maycause the display section to display the information related to a targetselected from an object in a real space visually recognized through thedisplay section in a display form corresponding to the target.

According to the aspect with this configuration, it is possible to showthe information concerning the object in the real space visuallyrecognized through the display section to the person viewing the displaydevice as if the information corresponds to the object in the realspace.

In the display device according to the aspect, the control section maycause the display section to display the information for highlightingthe target in the real space visually recognized through the displaysection to be superimposed on the target.

According to the aspect with this configuration, it is possible to showthe object in the real space to the person viewing the display devicewhile highlighting the object.

In the display device according to the aspect, the control section mayswitch, according to the input recognized by the input section, theinformation displayed in the display form corresponding to the target tothe information corresponding to a target different from the target anddisplay the information.

According to the aspect with this configuration, it is possible tochange the target, the information of which is displayed, according tothe input and switch and display the information. Therefore, it ispossible to sequentially display different information corresponding toa different target.

In the display device according to the aspect, the input section maydetect an input in a range visually recognized through the displaysection and specify an input position, and the control section maychange the display in the display section according to the inputposition specified by the input section.

According to the aspect with this configuration, when the inputrecognizable by the input section is performed, it is possible to changethe display according to the input position.

In the display device according to the aspect, the control section maycause the display section to display an indicator to be superimposed onthe item being displayed in a position corresponding to the inputposition specified by the input section.

According to the aspect with this configuration, it is possible tochange, by performing the input, a display form of an item to bedisplayed.

In the display device according to the aspect, the display device mayfurther include a state detecting section configured to detect anexecution state of the motion corresponding to the item on the basis ofa picked-up image of the outside scene or voice collected from theoutside scene, and the control section may change, according to theexecution state detected by the state detecting section, a display stateof the item being displayed by the display section.

According to the aspect with this configuration, it is possible todetect the execution state of the motion and change the display stateaccording to the detected execution state.

In the display device according to the aspect, the display device mayfurther include an execution-data storing section configured to storeexecution data including information related to the execution state ofthe motion, and the control section may cause the execution-data storingsection to store, in association with the item data, execution data ofthe motion corresponding to the item included in the item data and, whenthe execution data is read out from the execution-data storing section,control, on the basis of the read-out execution data, the display stateof the item displayed by the display section.

According to the aspect with this configuration, by storing dataconcerning the execution state of the motion and reading out the dataand reflecting the data on display, it is possible to perform recordingconcerning the execution state and readout of the recorded executionstate.

In the display device according to the aspect, the display device mayfurther include a target detecting section configured to detect a targetin the outside scene, and the control section may cause the displaysection to display the item included in the item data to correspond to aposition where the target detected by the target detecting section isvisually recognized through the display section.

According to the aspect with this configuration, it is possible toeasily visually recognize both of the place or the object, which is thetarget of the motion, and the item.

In the display device according to the aspect, the display device mayfurther include a communication section configured to communicate withan external apparatus and receive and acquire, with the communicationsection, the item data from the external apparatus.

According to the aspect with this configuration, it is possible toacquire the data concerning the item from the outside.

In the display device according to the aspect, the communication sectionmay execute near field radio communication.

According to the aspect with this configuration, it is possible toacquire the data concerning the item from the outside through the nearfield radio communication. Since a distance in which the near fieldradio communication can be performed is limited, it is possible toacquire data associated with a place.

A control method for a control device according to another aspect of theinvention includes a display section configured to display an image tobe visually recognizable through an outside scene, the control methodincluding: acquiring data of a list including a plurality of items;detecting an input in a range visually recognized through the displaysection and specifying an input position; and causing, on the basis ofitem data including an item, the display section to display the item andchange a display state of the item being displayed in a positioncorresponding to the specified input position.

According to the aspect, a person viewing the display device can changethe display of the item by viewing an item to be displayed andperforming an input recognizable by the input section. Consequently, forexample, a person who performs a motion targeting a place or an objectcan view the item with the display section and visually recognize theplace or the object, which is the target of the motion, through thedisplay section. In this way, the person can view the target present ina real space, which is an outside scene, and the item even if the persondoes not perform a movement with a large load such as a large movementof a visual line and can change the display.

A computer program according to still another aspect of the invention isexecutable by a computer that controls a display device including adisplay section configured to display an image to be visuallyrecognizable through an outside scene, the computer program causing thecomputer to: detect an input in a range visually recognized through thedisplay section and specify an input position; and cause, on the basisof item data including an item, the display section to display the itemand change a display state of the item being displayed in a positioncorresponding to the specified input position.

A storage medium according to yet another aspect of the invention hasthe computer program stored therein.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be described with reference to the accompanyingdrawings, wherein like numbers reference like elements.

FIG. 1 is an explanatory diagram showing the exterior configuration ofan HMD in a first embodiment.

FIG. 2 is a diagram showing the configuration of optical systems of animage display section.

FIGS. 3A and 3B are diagrams showing the main part configuration of theimage display section.

FIG. 4 is a functional block diagram of sections included in the HMD.

FIG. 5A is a schematic diagram showing a configuration example of datastored by a storing section and shows a configuration example ofscenario data and history data.

FIG. 5B is a schematic diagram showing a configuration example of thedata stored by the storing section and shows another configurationexample of the scenario data and the history data.

FIG. 6A is an explanatory diagram of an input method in the HMD andshows an input example in which a virtual keyboard is used.

FIG. 6B is an explanatory diagram of an input method in the HMD andshows an example of a position input.

FIG. 7A is a diagram showing a display example in the HMD.

FIG. 7B is a diagram showing a display example in the HMD.

FIG. 7C is a diagram showing a display example in the HMD.

FIG. 8A is a schematic diagram showing another configuration example ofthe data stored by the storing section and shows a configuration exampleof scenario data.

FIG. 8B is a schematic diagram showing another configuration example ofthe data stored by the storing section and shows a configuration exampleof display setting data.

FIG. 9A is a diagram showing a display example in the HMD.

FIG. 9B is a diagram showing a display example in the HMD.

FIG. 9C is a diagram showing a display example in the HMD.

FIG. 9D is a diagram showing a display example in the HMD.

FIG. 10A is a diagram showing a display example in the HMD.

FIG. 10B is a diagram showing a display example in the HMD.

FIG. 10C is a diagram showing a display example in the HMD.

FIG. 10D is a diagram showing a display example in the HMD.

FIG. 11 is a diagram showing a display example in the HMD.

FIG. 12A is a diagram showing a display example in the HMD.

FIG. 12B is a diagram showing a display example in the HMD.

FIG. 13 is a flowchart for explaining the operation of the HMD in thefirst embodiment.

FIG. 14 is a flowchart for explaining the operation of an HMD in asecond embodiment.

DESCRIPTION OF EXEMPLARY EMBODIMENTS First Embodiment

FIG. 1 is an explanatory diagram showing the exterior configuration ofan HMD (Head Mounted Display) 100 according to an embodiment appliedwith the invention.

The HMD 100 is a display device including an image display section 20that causes a user to visually recognize a virtual image in a state inwhich the image display section 20 is worn on the head of the user and acontrol device 10 that controls the image display section 20. Thecontrol device 10 also functions as a controller with which the usercontrols the HMD 100.

The image display section 20 is a wearing body worn on the head of theuser. In this embodiment, the image display section 20 has an eyeglassshape. The image display section 20 includes a right holding section 21,a right display driving section 22, a left holding section 23, a leftdisplay driving section 24, a right optical-image display section 26, aleft optical-image display section 28, a right camera 61 (an imagepickup section), a left camera 62 (an image pickup section), and amicrophone 63. The right optical-image display section 26 and the leftoptical-image display section 28 are disposed to be respectively locatedin front of the right eye and in front of the left eye of the user whenthe user wears the image display section 20. One end of the rightoptical-image display section 26 and one end of the left optical-imagedisplay section 28 are connected to each other in a positioncorresponding to the middle of the forehead of the user when the userwears the image display section 20.

The right holding section 21 is a member provided to extend from an endportion ER, which is the other end of the right optical-image displaysection 26, to a position corresponding to the temporal region of theuser when the user wears the image display section 20. Similarly, theleft holding section 23 is a member provided to extend from an endportion EL, which is the other end of the left optical-image displaysection 28, to a position corresponding to the temporal region of theuser when the user wears the image display section 20. The right holdingsection 21 and the left holding section 23 hold the image displaysection 20 on the head of the user like temples of eyeglasses.

The right display driving section 22 and the left display drivingsection 24 are disposed on sides opposed to the head of the user whenthe user wears the image display section 20. Note that, in the followingexplanation, the right display driving section 22 and the left displaydriving section 24 are collectively simply referred to as “displaydriving sections” as well and the right optical-image display section 26and the left optical-image display section 28 are collectively simplyreferred to as “optical-image display sections” as well.

The display driving sections 22 and 24 include liquid crystal displays241 and 242 (hereinafter referred to as “LCDs 241 and 242” as well) andprojection optical systems 251 and 252 explained below with reference toFIGS. 2 to 4.

The right optical-image display section 26 and the left optical-imagedisplay section 28 include light guide plates 261 and 262 (FIG. 2) anddimming plates 20A. The light guide plates 261 and 262 are formed of alight transmissive resin or the like and guide image lights output bythe display driving sections 22 and 24 to the eyes of the user. Thedimming plates 20A are thin plate-like optical elements and are arrangedto cover the front side of the image display section 20 on the oppositeside of the side of the eyes of the user. As the dimming plates 20A,various dimming plates such as a dimming plate having almost no lighttransmissivity, a dimming plate nearly transparent, a dimming plate thatattenuates a light amount and transmits light, and a dimming plate thatattenuates or reflects light having a specific wavelength can be used.By selecting optical characteristics (light transmittance, etc.) of thedimming plates 20A as appropriate, it is possible to adjust an amount ofexternal light made incident on the right optical-image display section26 and the left optical-image display section 28 from the outside andadjust easiness of visual recognition of a virtual image. In thisembodiment, the dimming plates 20A at least having light transmissivityenough for enabling the user wearing the HMD 100 to visually recognizean outside scene are used. The dimming plates 20A protect the rightlight guide plate 261 and the left light guide plate 262 and suppressdamage, adhesion of stain, and the like to the right light guide plate261 and the left light guide plate 262.

The dimming plates 20A may be detachably attachable to the rightoptical-image display section 26 and the left optical-image displaysection 28. A plurality of kinds of dimming plates 20A may bereplaceable and attachable. The dimming plates 20A may be omitted.

The right camera 61 is disposed at the end portion on the right holdingsection 21 side on the front surface of the HMD 100. The left camera 62is disposed at the end portion on the left holding section 23 side onthe front surface of the HMD 100. The right camera 61 and the leftcamera 62 are digital cameras including image pickup devices such asCCDs or CMOSs and image pickup lenses. The right camera 61 and the leftcamera 62 may be a monocular camera or may be a stereo camera.

The right camera 61 and the left camera 62 pick up images of at least apart of an outside scene in a front side direction of the HMD 100, inother words, in a visual field direction of the user in a state in whichthe HMD 100 is mounted. In another expression, at least one of the rightcamera 61 and the left camera 62 picks up an image in a range ordirection overlapping a visual field of the user. More specifically, atleast one of the right camera 61 and the left camera 62 picks up animage in a direction gazed by the user. The breadth of angles of view ofthe right camera 61 and the left camera 62 can be set as appropriate. Inthis embodiment, as explained below, the angles of view of the rightcamera 61 and the left camera 62 are angles of view including an outsideworld that the user visually recognizes through the right optical-imagedisplay section 26 and the left optical-image display section 28.Further, it is more desirable that an image pickup range of the rightcamera 61 and the left camera 62 is set such that the right camera 61and the left camera 62 can pick up an image of the entire visual fieldof the user through the dimming plates 20A.

The right camera 61 and the left camera 62 execute image pickupaccording to control by an image-pickup control section 161 (FIG. 4)included in a control section 140 and output picked-up image data to theimage-pickup control section 161.

The HMD 100 may include a distance sensor (not shown in the figure) thatdetects a distance to a measurement target object located in ameasurement direction set in advance. The distance sensor is disposedin, for example, a boundary portion between the right optical-imagedisplay section 26 and the left optical-image display section 28. Inthis case, in a state in which the user wears the image display section20, the position of the distance sensor is substantially the middle ofboth the eyes of the user in the horizontal direction and is above boththe eyes of the user in the vertical direction. The measurementdirection of the distance sensor can be set in, for example, a directionoverlapping an image pickup direction of the right camera 61 and theleft camera 62 in the front side direction of the HMD 100.

The distance sensor can include a light source such as an LED or a laserdiode and a light receiving section that receives reflected light oflight emitted by the light source and reflected on a measurement targetobject. The distance sensor only has to execute triangular range findingprocessing and range finding processing based on a time differenceaccording to control by the control section 140. The distance sensor mayinclude a sound source that emits ultrasound and a detecting sectionthat receives the ultrasound reflected on the measurement target object.In this case, the distance sensor only has to execute the range findingprocessing on the basis of a time difference until the reflection of theultrasound according to the control by the control section 140.

FIG. 2 is a main part plan view showing the configuration of opticalsystems included in the image display section 20. A left eye LE and aright eye RE of the user are shown in FIG. 2 for explanation.

The left display driving section 24 includes a left backlight 222including a light source such as an LED and a diffuser, the left LCD 242of a transmission type disposed on an optical path of light diffused bythe diffuser of the left backlight 222, and a left projection opticalsystem 252 including a lens group that guides image light L transmittedthrough the left LCD 242. The left. LCD 242 is a transmission-typeliquid crystal panel on which a plurality of pixels are arranged in amatrix shape.

The left projection optical system 252 includes a collimate lens thatchanges the image light L emitted from the left. LCD 242 to light beamsin a parallel state. The image light L changed to the light beams in theparallel state by the collimate lens is made incident on the left lightguide plate 262. The left light guide plate 262 is a prism on which aplurality of reflection surfaces for reflecting the image light L areformed. The image light L is guided to the left eye LE side through aplurality of times of reflection on the inside of the left light guideplate 262. On the left light guide plate 262, a half mirror 262A (areflection surface) located in front of the left eye LE is formed.

The image light L reflected on the half mirror 262A is emitted from theleft optical-image display section 28 toward the left eye LE. The imagelight L forms an image on the retina of the left eye LE and causes theuser to visually recognize the image.

The right display driving section 22 is configured symmetrically to theleft display driving section 24. The right display driving section 22includes a right backlight 221 including a light source such as an LEDand a diffuser, the right LCD 241 of the transmission type disposed onan optical path of light diffused from the diffuser of the rightbacklight 221, and the right projection optical system 251 including alens group that guides the image light L transmitted through the rightLCD 241. The right LCD 241 is a transmission-type liquid crystal panelon which a plurality of pixels are arranged in a matrix shape.

The right projection optical system 251 includes a collimate lens thatchanges the image light L emitted from the right LCD 241 to light beamsin a parallel state. The image light L changed to the light beams in theparallel state by the collimate lens is made incident on the right lightguide plate 261 (an optical element). The right light guide plate 261 isa prism on which a plurality of reflection surfaces for reflecting theimage light L are formed. The image light L is guided to the right eyeRE side through a plurality of times of reflection on the inside of theright light guide plate 261. On the right light guide plate 261, a halfmirror 261A (a reflection surface) located in front of the right eye REis formed.

The image light L reflected on the half mirror 261A is emitted from theright optical-image display section 26 toward the right eye RE. Theimage light L forms an image on the retina of the right eye RE andcauses the user to visually recognize the image.

The image light L reflected on the half mirror 261A and external lightOL transmitted through the dimming plate 20A are made incident on theright eye RE of the user. The image light 1, reflected on the halfmirror 262A and the external light OL transmitted through the dimmingplate 20A are made incident on the left eye LE. In this way, the HMD 100makes the image light L of the image processed on the inside and theexternal light OL incident on the eyes of the user to be superimposedone on top of the other. For the user, the outside scene is seen throughthe dimming plates 20A. The image formed by the image light L isvisually recognized to be superimposed the outside scene. In this way,the HMD 100 functions as a see-through type display device.

Note that the left projection optical system 252 and the left lightguide plate 262 are collectively referred to as “left light guidesection” as well. The right projection optical system 251 and the rightlight guide plate 261 are collectively referred to as “right light guidesection” as well. The configuration of the right light guide section andthe left light guide section is not limited to the example explainedabove. Any system can be used as long as a virtual image is formed infront of the eyes of the user using image light. For example, adiffraction grating may be used or a transreflective film may be used.

The image display section 20 is connected to the control device 10 via aconnecting section 40. The connecting section 40 includes a main bodycord 48 connected to the control device 10, a right cord 42, a left cord44, and a coupling member 46. The right cord 42 and the left cord 44 aretwo cords branching from the main body cord 48. The right cord 42 isinserted into a housing of the right holding section 21 from a distalend portion AP in an extending direction of the right holding section 21and connected to the right display driving section 22. Similarly, theleft cord 44 is inserted into a housing of the left holding section 23from a distal end portion AP in an extending direction of the leftholding section 23 and connected to the left display driving section 24.

The coupling member 46 is provided at a branching point of the main bodycord 48 and the right and left cords 42 and 44 and includes a jack forconnecting an earphone plug 30. A right earphone 32 and a left earphone34 extend from the earphone plug 30. The microphone 63 is provided inthe vicinity of the earphone plug 30. Cords between the earphone plug 30and the microphone 63 are collected as one cord. Cords branch from themicrophone 63 and are respectively connected to the right earphone 32and the left earphone 34.

For example, as shown in FIG. 1, the microphone 63 is disposed to directa sound collecting section of the microphone 63 to the visual linedirection of the user. The microphone 63 collects sound and outputs asound signal to a sound processing section 187 (FIG. 4). The microphone63 may be, for example, a monaural microphone or a stereo microphone,may be a microphone having directivity, or may be a nondirectionalmicrophone.

The right cord 42, the left cord 44, and the main body cord 48 only haveto be cords capable of transmitting digital data and can be configuredby, for example, a metal cable or an optical fiber. The right cord 42and the left cord 44 may be collected as one cord.

The image display section 20 and the control device 10 transmit varioussignals via the connecting section 40. Connectors (not shown in thefigure), which fit with each other, are provided at an end of the mainbody cord 48 on the opposite side of the coupling member 46 and in thecontrol device 10. The control device 10 and the image display section20 can be connected and disconnected by fitting and unfitting of theconnector of the main body cord 48 and the connector of the controldevice 10.

The control device 10 controls the HMD 100. The control device 10includes a determination key 11, a lighting section 12, a displayswitching key 13, a luminance switching key 15, a direction key 16, amenu key 17, and switches including a power switch 18. The controldevice 10 also includes a track pad 14 operated by the user withfingers.

The determination key 11 detects pressing operation and outputs a signalfor determining content of the operation in the control device 10. Thelighting section 12 includes a light source such as an LED (LightEmitting Diode) and notifies, with a lighting state of the light source,an operation state of the HMD 100 (e.g., ON/OFF of a power supply). Thedisplay switching key 13 outputs, according to pressing operation, forexample, a signal for instructing switching of a display mode of animage.

The track pad 14 includes an operation surface for detecting contactoperation and outputs an operation signal according to operation on theoperation surface. A detection type on the operation surface is notlimited. An electrostatic type, a pressure detection type, an opticaltype, and the like can be adopted. The luminance switching key 15outputs, according to pressing operation, a signal for instructing anincrease and a decrease in the luminance of the image display section20. The direction key 16 outputs an operation signal according topressing operation on keys corresponding to the upward, downward, left,and right directions. The power switch 18 is a switch that switchespower ON/OFF of the HMD 100.

FIGS. 3A and 3B are diagrams showing the main part configuration of theimage display section 20. FIG. 3A is a main part perspective view of theimage display section 20 viewed from the head side of the user. FIG. 3Bis an explanatory diagram of angles of view of the right camera 61 andthe left camera 62. Note that, in FIG. 3A, the right cord 42, the leftcord 44, and the like connected to the image display section 20 are notshown.

FIG. 3A is a side in contact with the head of the user of the imagedisplay section 20, in other words, a side seen by a right eye RE and aleft eye LE of the user. In other words, the rear sides of the rightoptical-image display section 26 and the left optical-image displaysection 28 are seen.

In an example shown in FIG. 3A, the half mirror 261A for radiating imagelight on the right eye RE of the user and the half mirror 262A forradiating image light on the left eye LE of the user are seen assubstantially square regions. The entire right and left optical-imagedisplay sections 26 and 28 including the half mirrors 261A and 262Atransmit external light as explained above. Therefore, the user visuallyrecognizes an outside scene through the entire right and leftoptical-image display sections 26 and 28 and visually recognizesrectangular display images in the positions of the half mirrors 261A and262A.

As explained above, the right camera 61 is disposed at the end portionon the right holding section 21 side to face the front of the imagedisplay section 20. The left camera 62 is disposed at the end portion onthe left holding section 23 side.

FIG. 3B is a diagram schematically showing, in plan view, the positionsof the right camera 61 and the left camera 62 together with the righteye RE and the left eye LE. An angle of view (an image pickup range) ofthe right camera 61 is indicated by CR. An angle of view (an imagepickup range) of the left camera 62 is indicated by CL. Note that, inFIG. 3B, the angles of view CR and CL in the horizontal direction areshown. However, actual angles of view of the right camera 61 and theleft camera 62 expand in the up-down direction like an angle of view ofa general digital camera.

The angle of view CR and the angle of view CL are substantiallysymmetrical with respect to the center position of the image displaysection 20. Both of the angle of view CR and the angle of view CLinclude the right front direction in the center position of the imagedisplay section 20. Therefore, the angles of view CR and CL overlap inthe front in the center position of the image display section 20.

For example, as shown in FIG. 3B, when a target OB is present in thefront direction of the image display section 20, the target OB isincluded in both of the angle of view CR and the angle of view CL.Therefore, the target OB appears in both of a picked-up image of theright camera 61 and a picked-up image of the left camera 62. When theuser gazes at the target OB, the visual line of the user is directed tothe target OB as indicated by signs RD and LD in the figure. In general,a viewing angle of a human is approximately 200 degrees in thehorizontal direction and approximately 125 degrees in the verticaldirection. In the viewing angle, an effective visual field excellent ininformation acceptability is approximately 30 degrees in the horizontaldirection and approximately 20 degrees in the vertical direction.Further, a stable gazing field in which a gazing point of the human isquickly and stably seen is approximately 60 to 90 degrees in thehorizontal direction and approximately 45 to 70 degrees in the verticaldirection.

Therefore, when the gazing point is the target OB, the effective visualfield is approximately 30 degrees in the horizontal direction andapproximately 20 degrees in the vertical direction centering on thevisual lines RD and LD. The stable gazing field is approximately 60 to90 degrees in the horizontal direction and approximately 45 to 70degrees in the vertical direction. The viewing angle is approximately200 degrees in the horizontal direction and approximately 125 degrees inthe vertical direction.

An actual visual field visually recognized by the user wearing the HMD100 through the right optical-image display section 26 and the leftoptical-image display section 28 of the image display section 20 isreferred to as an actual visual field (FOV). In the configuration ofthis embodiment shown in FIGS. 1 and 2, the actual visual field isequivalent to an actual visual field visually recognized by the userthrough the right optical-image display section 26 and the leftoptical-image display section 28. The actual visual field is narrowerthan the viewing angle and the stable gazing field explained withreference to FIG. 3B but is wider than the effective visual field.

The right camera 61 and the left camera 62 are desirably capable ofpicking up images in a range wider than the visual field of the user.Specifically, the entire angles of view CR and CL are desirably widerthan at least the effective visual field of the user. The entire anglesof view CR and CL are more desirably wider than the actual visual fieldof the user. The entire angles of view CR and CL are still moredesirably wider than the stable gazing field of the user. The entireangles of view CR and CL are most desirably wider than the viewing angleof the user.

Therefore, in the right camera 61 and the left camera 62, the angle ofview CR and the angle of view CL are arranged to overlap in the front ofthe image display section 20 as shown in FIG. 3B. The right camera 61and the left camera 62 may be configured by wide-angle cameras. That is,the right camera 61 and the left camera 62 may include so-calledwide-angle lenses as image pickup lenses and may be capable of pickingup images in a wide angle of view. The wide-angle lens may includelenses called super-wide-angle lens and semi-wide-angle lens. Thewide-angle lens may be a single focus lens or may be a zoom lens. Theright camera 61 and the left camera 62 may include a lens groupconsisting of a plurality of lenses. The angle of view CR of the rightcamera 61 and the angle of view CL of the left camera 62 do not have tobe the same angle. An image pickup direction of the right camera 61 andan image pickup direction of the left camera 62 do not need to becompletely parallel. When a picked-up image of the right camera 61 and apicked-up image of the left camera 62 are superimposed, an image in arange wider than the visual field of the user only has to be picked up.

FIG. 4 is a functional block diagram of the sections included in the HMD100.

The HMD 100 includes an interface 125 that connects various externalapparatuses OA functioning as supply sources of contents. As theinterface 125, interfaces adapted to wired connection such as a USBinterface, a micro USB interface, or an interface for a memory card canbe used. The interface 125 may be configured by a radio communicationinterface. The external apparatus OA is an image supply apparatus thatsupplies images to the HMD 100. As the external apparatus OS, forexample, a personal computer (PC), a cellular phone terminal, or aportable game machine is used.

The control device 10 includes the control section 140, aninput-information acquiring section 110, a storing section 120, and atransmitting section (Tx) 51 and a transmitting section (Tx) 52.

The input-information acquiring section 110 is connected to an operationsection 135. The operation section 135 includes the track pad 14, thedirection key 16, and the power switch 18. The input-informationacquiring section 110 acquires input content on the basis of a signalinput from the operation section 135. The control device 10 includes apower supply section (not shown in the figure) and supplies electricpower to the sections of the control device 10 and the image displaysection 20.

The storing section 120 is a nonvolatile storage device and has storedtherein various computer programs and data related to the computerprograms. The storing section 120 may store data of still images andmoving images to be displayed on the image display section 20.

The storing section 120 stores setting data 121. The setting data 121includes various setting values used by the control section 140. Thesetting values included in the setting data 121 may be values input inadvance by operation of the operation section 135. The storing section120 may receive setting values from the external apparatus OA or otherapparatuses (not shown in the figure) via the communication section 117or the interface 125 and store the setting values.

The storing section 120 stores scenario data 123 and history data 124.The scenario data 123 is data used for informing the user wearing theHMD 100 of a work procedure and the like. The history data 124 is dataincluding information concerning the progress of work, an executionhistory, and the like related to the scenario data 123. The scenariodata 123 and the history data 124 are explained below. The scenario data123 includes item data and order data. The storing section 120 isequivalent to the order-data storing section. Since the storing section120 stores the history data 124 (execution data), the storing section120 is equivalent to the execution-data storing section.

A three-axis sensor 113, a GPS 115, a communication section 117, anear-field-radio communication section 119, and a sound recognizingsection 114 are connected to the control section 140. The three-axissensor 113 is a three-axis acceleration sensor. The control section 140acquires detection values of the three-axis sensor 113. With thethree-axis sensor 113, the control section 140 can detect a movement ofthe control device 10 and can detect operation such as shaking of thecontrol device 10. The three-axis sensor 113 may be replaced with anine-axis sensor including a three-axis acceleration sensor, athree-axis angular velocity sensor, and a three-axis terrestrialmagnetism sensor. In this case, the control section 140 can acquirerespective detection values of the three-axis acceleration sensor, thethree-axis angular velocity sensor, and the three-axis terrestrialmagnetism sensor and detect, for example, a posture, a direction, and amovement of the control device 10.

The GPS 115 includes an antenna (not shown in the figure), receives aGPS (Global Positioning System) signal, and calculates the presentposition of the control device 10. The GPS 115 outputs the presentposition and the present time calculated on the basis of the GPS signalto the control section 140. The GPS 115 may include a function ofacquiring the present time on the basis of information included in theGPS signal and causing the control section 140 to correct time clockedby the control section 140.

The communication section 117 executes wireless data communicationconforming to a standard of wireless communication such as a wirelessLAN (Win (registered trademark)), a Miracast (registered trademark), ora Bluetooth (registered trademark).

When the external apparatus OA is connected to the communication section117 by radio, the control section 140 acquires content data from thecommunication section 117 and causes the image display section 20 todisplay an image. On the other hand, when the external apparatus OA isconnected to the interface 125 by wire, the control section 140 acquirescontent data from the interface 125 and causes the image display section20 to display an image. The communication section 117 and the interface125 function as a data acquiring section DA that acquires content datafrom the external apparatus OA.

The near-field-radio communication section 119 is a communicationsection that executes near field radio data communication according tothe control by the control section 140. The near-field-radiocommunication section 119 includes a radio communication interfaceincluding an antenna, an RF circuit, and a baseband circuit not shown inthe figure. The near-field-radio communication section 119 executes, forexample, radio communication called NFC (Near Field radioCommunication). Specifically, the near-field-radio communication section119 performs near field radio communication such as Felica (registeredtrademark), ISO/IEC 14443, and ISO/IEC 18092. The near-field-radiocommunication section 119 may execute radio communication conforming tothe Bluetooth standard. In this case, for example, the near-field-radiocommunication section 119 desirably includes an antenna and an RFcircuit (not shown in the figure) corresponding to a Bluetooth class 3(with an output of 1 mw) and is suitable for radio communication in ashort distance. The near-field-radio communication section 119 may beconfigured as a so-called IC tag reader, execute radio communicationwith an external IC tag (not shown in the figure), and acquire data fromthe IC tag.

The control section 140 includes a CPU (not shown in the figure) thatexecutes a computer program, a RAM (not shown in the figure) thattemporarily stores the computer program executed by the CPU and data,and a ROM (not shown in the figure) that stores, in a nonvolatilemanner, a basic control program executed by the CPU and data. Thecontrol section 140 reads out and executes a computer program stored bythe storing section 120 and functions as an operating system (OS) 150,an image processing section 160, the image-pickup control section 161, aposition detecting section 162, an AR-display control section 163, astate detecting section 164, a communication control section 165, asound processing section 187, and a display control section 190.

The image processing section 160 acquires an image signal included incontents. The image processing section 160 separates, from the acquiredimage signal, synchronization signals such as a vertical synchronizationsignal VSync and a horizontal synchronization signal HSync. The imageprocessing section 160 generates, according to cycles of the verticalsynchronization signal VSyne and the horizontal synchronization signalHSync separated from the image signal, a clock signal PCLK using a PLL(Phase Locked Loop) circuit or the like (not shown in the figure). Theimage processing section 160 converts an analog image signal, from whichthe synchronization signals are separated, into a digital image signalusing an A/D conversion circuit or the like (not shown in the figure).The image processing section 160 stores the digital image signal afterthe conversion in the RAM of the control section 140 frame by frame asimage data (in the figure, Data) of a target image. The image data is,for example, RUB data.

Note that the image processing section 160 may perform, according tonecessity, resolution conversion processing for converting theresolution of the image data into resolution suitable for the rightdisplay driving section 22 and the left display driving section 24. Theimage processing section 160 may execute, for example, image adjustmentprocessing for adjusting the luminance and the chroma of the image dataand 2D/3D conversion processing for generating 2D image data from 3Dimage data or generating 3D image data from 2D image data.

The image processing section 160 transmits the clock signal PCLK, thevertical synchronization signal Vsync, the horizontal synchronizationsignal HSync, and the image data Data stored in the RAM via thetransmitting sections 51 and 52. The transmitting sections 51 and 52function as a transceiver and execute serial transmission between thecontrol device 10 and the image display section 20. Note that the imagedata Data transmitted via the transmitting section 51 is referred to as“image data for right eye” and the image data Data transmitted via thetransmitting section 52 is referred to as “image data for left eye”.

The display control section 190 generates a control signal forcontrolling the right display driving section 22 and the left displaydriving section 24 and controls, with the control signal, generation andemission of image light by each of the right display driving section 22and the left display driving section 24. Specifically, the displaycontrol section 190 controls driving ON/OFF of the right LCD 241 by aright LCD control section 211 and driving ON/OFF of the right backlight221 by a right backlight control section 201. The display controlsection 190 controls driving ON/OFF of the left LCD 242 by a left LCDcontrol section 212 and driving ON/OF of the left backlight 222 by aleft backlight control section 202.

The sound processing section 187 acquires a sound signal included incontents, amplifies the acquired sound signal, and outputs the amplifiedsound signal to the right earphone 32 and the left earphone 34. Thesound processing section 187 acquires sound collected by the microphone63 and converts the sound into digital sound data. The sound processingsection 187 may perform processing set in advance on the digital sounddata.

The image display section 20 includes the right camera 61 and the leftcamera 62. The image display section 20 includes an interface 25, theright display driving section 22, the left display driving section 24,the right light guide plate 261 functioning as the right optical-imagedisplay section 26, the left light guide plate 262 functioning as theleft optical-image display section 28, and a nine-axis sensor 66.

The nine-axis sensor 66 is a motion sensor that detects acceleration(three axes), angular velocity (three axes), and terrestrial magnetism(three axes). When the image display section 20 is worn on the head ofthe user, the control section 140 can detect a movement of the head ofthe user on the basis of detection values of the nine-axis sensor 66.For example, the control section 140 can estimate the magnitude and thedirection of a tilt of the image display section 20 on the basis of thedetection values of the nine-axis sensor 66.

The interface 25 includes a connector to which the right cord 42 and theleft cord 44 are connected. The interface 25 outputs the clock signalPCLK, the vertical synchronization signal VSync, the horizontalsynchronization signal HSync, and the image data Data transmitted fromthe transmitting section 51 to a receiving section (Rx) 53 or 54corresponding to the transmitting section 51. The interface 25 outputsthe control signal transmitted from the display control section 190 tothe receiving section 53 or 54 and the right backlight control section201 or the left backlight control section 202 corresponding to thedisplay control section 190.

The interface 25 is an interface that connects the right camera 61, theleft camera 62, and the nine-axis sensor 66. Picked-up image data of theright camera 61 and the left camera 62 and detection results ofacceleration (three axes), angular velocity (three axes), andterrestrial magnetism (three axes) by the nine-axis sensor 66 are sentto the control section 140 via the interface 25.

The right display driving section 22 includes the right backlight 221,the right LCD 241, and the right projection optical system 251. Theright display driving section 22 includes the receiving section 53, theright backlight (BL) control section 201 that controls the rightbacklight (BL) 221, and the right LCD control section 211 drives theright LCD 241.

The receiving section 53 operates as a receiver corresponding to thetransmitting section 51 and executes serial transmission between thecontrol device 10 and the image display section 20. The right backlightcontrol section 201 drives the right backlight 221 on the basis of aninput control signal. The right LCD control section 211 drives the rightLCD 241 on the basis of the clock signal PCLK, the verticalsynchronization signal VSync, the horizontal synchronization signalHSync, and the image data for right eye Data input via the receivingsection 53.

The left display driving section 24 has a configuration same as theconfiguration of the right display driving section 22. The left displaydriving section 24 includes the left backlight 222, the left LCD 242,and the left projection optical system 252. The left display drivingsection 24 includes the receiving section 54, the left backlight controlsection 202 that drives the left backlight 222, and the left LCD controlsection 212 that drives the left LCD 242.

The receiving section 54 operates as a receiver corresponding to thetransmitting section 52 and executes serial transmission between thecontrol device 10 and the image display section 20. The left backlightcontrol section 202 drives the left backlight 222 on the basis of aninput control signal. The left LCD control section 212 drives the leftLCD 242 on the basis of the clock signal PCLK, the verticalsynchronization signal VSync, the horizontal synchronization signalHSync, and the image data for right eye Data input via the receivingsection 54.

Note that the right backlight control section 201, the right LCD controlsection 211, the right backlight 221, and the right LCD 241 arecollectively referred to as “image-light generating section” on theright as well. Similarly, the left backlight control section 202, theleft LCD control section 212, the left backlight 222, and the left LCD242 are collectively referred to as “image-light generating section” onthe left as well.

As shown in FIG. 3A, a range in which the user wearing the HMD 100visually recognizes image lights radiated from the half mirrors 261A and262A is smaller than the actual visual field. For example, when the HMD100 displays an image that achieves an AR (Augmented Reality) effect(hereinafter referred to as AR image), the HMD 100 displays the AR imageto overlap the target OB gazed by the user. In this case, the AR effectis obtained because the AR image is seen overlapping the target OB.However, a region where the AR image is visually recognized is limitedby the size of the half mirrors 261A and 262A. Therefore, even when thetarget OB is included in the actual visual field, it is sometimesdifficult to display the AR image in a position overlapping the targetOB. When an image other than the AR image is displayed, a region wherethe HMD 100 can display the image is smaller than the actual visualfield. This is likely to limit functions.

The HMD 100 performs display corresponding to the outside scene, forexample, displays AR contents corresponding to the target OB in theoutside scene gazed by the user using images of the outside scene in arange wider than the actual visual field of the user such as picked-upimages of the right camera 61 and the left camera 62 and images preparedin advance.

In the HMD 100, the half mirror 261A and the half mirror 262A reflectimage lights on the eyes of the user to thereby form a display region.The HMD 100 reflects the image lights with the half mirrors 261A and261B and causes the eyes of the user to visually recognize virtualimages. Therefore, in this embodiment, the display region is not thehalf mirrors 261A and 262A themselves and is a region where the usersenses the image lights reflected on the half mirrors 261A and 262A. Notthat, when the image display section 20 causes the half mirrors 261A and262A to form images, the half mirrors 261A and 262A are equivalent tothe display region.

The display region is a region corresponding to the right LCD 241 andthe left LCD 242. In the display region, the user visually recognizesimages displayed on the LCDs 241 and 242. For example, when images aredisplayed over entire regions where the right LCD 241 and the left LCD242 can display images (a displayable region), it is possible to causethe user to visually recognize an image having the size of the entiredisplay region.

For example, as explained below, the display region is locatedsubstantially in the center of the visual field of the user and is aslarge as the visual field or smaller than the visual field.

The image-pickup control section 161 controls the right camera 61 andthe left camera 62 to execute image pickup and acquires picked-up imagedata. The image-pickup control section 161 may cause one of the rightcamera 61 and the left camera 62 to execute the image pickup or maycause both of the right camera 61 and the left camera 62 to execute theimage pickup.

The position detecting section 162 (the input section and the targetdetecting section) executes a function of the control section 140detecting (recognizing) a target (a target object) from an outside sceneimage. In the function, the position detecting section 162 analyzes, forexample, picked-up image data acquired by the image-pickup controlsection 161. Specifically, the position detecting section 162 detectsthe position of a target of AR display and an operation position ofoperation by the user on the basis of the picked-up image data of theright camera 61 and/or the left camera 62. In the position detectingsection 162, as a target to be detected, a target on which the ARdisplay is performed and a pointer used for operation by the user areset.

The target of the AR display forms a part of a range visually recognizedby the user through the image display section 20 (the visual field ofthe user) such as an object in a real space (including a fixed objectsuch as a building) or a scene. The target of the AR display is detectedfrom the picked-up image data of the right camera 61 and the left camera62. The control section 140 displays AR contents according to thedetected target.

The target can be set for each of contents of the AR display. Forexample, data used in processing for detecting a target may be includedin data of the contents of the AR display (hereinafter referred to as ARcontents). Data concerning a target to be detected may be stored as thesetting data 121. The position detecting section 162 acquires data fordetecting a target from the AR contents or the setting data 121 andrecognizes a target included in an outside scene image using the data.The data for detecting a target is data used for processing fordetecting an image of a target, which is an object present in the realspace, from a picked-up image and is, for example, a feature value ofthe image of the target. For example, when the target is an object, afeature value indicating a color, a shape, or another feature of apicked-up image of the object is included in the setting data. In thiscase, the position detecting section 162 performs processing forextracting an image of the object from image data of the outside sceneimage, calculates a feature value of the extracted image of the object,and compares and collates the calculated the feature value and thefeature value included in the setting data 121. When the feature valuesare values close to each other or the same value, the object of theimage extracted from the outside scene image can be recognized as atarget. When a plurality of feature values are included in the settingdata 121 concerning the target, the position detecting section 162 candetect the target from the outside image on the basis of the pluralityof feature values and recognize the target. When the position detectingsection 162 cannot recognize the target in the outside scene image, theposition detecting section 162 stays on standby until the image-pickupcontrol section 161 acquires a new outside scene image in step S12. Thedetecting section 162 performs processing for recognizing a targetconcerning the new outside scene image.

The control section 140 has a function of recognizing and detecting aninput by the user. The function can be realized as a function of theinput section. The input section detects an image of a pointer such as ahand of the user from picked-up images of the right camera 61 and/or theleft camera 62 and detects and recognizes a position, a direction, or amovement of the pointer as an input. For example, the input sectionspecifies the position of the pointer and detects a position input. Forexample, when determining that the movement of the pointer is equivalentto a gesture set in advance, the input section may detect and recognizea gesture input. The input section detects operation on the operationsection 135 on the basis of a signal input from the input-informationacquiring section 110. When the sound processing section 187 analyzesvoice collected by the microphone 63 and determines that the voice is avoice command set in advance, the input section may detect and recognizean input of the voice command. When determining that a pattern of adetection value of the nine-axis sensor 66 included in the image displaysection 20 corresponds to operation for knocking the image displaysection 20, the position detecting section 162 may detect and recognizethe knock operation on the image display section 20 as an input. Whendetecting an image of a marker for input such as a two-dimensional codeor a barcode from the picked-up images of the right camera 61 and/or theleft camera 62, the position detecting section 162 may detect andrecognize a marker reading input. When detecting an indicator or amarker from the picked-up image data of the right camera 61 and/or theleft camera 62, the position detecting section 162 may detect andrecognize a pointer or a marker corresponding to a visual line directionof the user detected by a visual line sensor 68.

The HMD 100 may include a footswitch (not shown in the figure) operatedby the user with a foot. The footswitch may be connected to the controlsection 140 or the operation section 135 by wire or may be connected tothe communication section 117 by radio communication such as theBluetooth (registered trademark). In this case, the input section maydetect operation of the footswitch and recognize the operation as aninput. Vital sensors such as an electromyograph (not shown in thefigure), a pulse measuring device (not shown in the figure), ablood-pressure measuring device (not shown in the figure), and ablood-oxygen-concentration measuring device (not shown in the figure)may be provided in the HMD 100. The input section may detect andrecognize an input on the basis of measurement values and detectionvalues of the vital sensors.

In this embodiment, the position detecting section 162 operates as theinput section. The position detecting section 162 detects an image ofthe pointer from the picked-up image data of the right camera 61 and theleft camera 62 and detects a position pointed by the pointer. Thepointer is an object used by the user for operation and is a part of thebody such as a hand or a finger of the user or an object manipulated bythe user. The shape of the object is not particularly limited and may beany shape such as a bar shape or a pen shape. A light emitting section(not shown in the figure) that emits infrared light or visible light maybe provided in the pointer. Marking including a design or a patterndetectable in the picked-up image data may be applied to the pointer. Inthis case, the position detecting section 162 can quickly performprocessing for detecting the pointer from the picked-up image data. Likethe processing for detecting the target from the picked-up image data,the position detecting section 162 detects the pointer from thepicked-up image data using the data of the feature values stored in thestoring section 120 in advance. The data of the feature values of thepointer is included in, for example, the setting data 121.

The position detecting section 162 specifies the position of the targetand the position of the pointer detected from the picked-up image data.Further, the position detecting section 162 performs processing forcalculating relative positions of the detected target and the pointerand the display region visually recognized by the user. Consequently, itis possible to perform association of the positions of images (includingcharacters, images, and signs) displayed by the right display drivingsection 22 and the left display driving section 24 and the positions ofthe target and the pointer in the real space. That is, it is possible toassociate positions where the user visually recognizes the imagesdisplayed by the right display driving section 22 and the left displaydriving section 24 and positions where the user visually recognizes thetarget and the pointer through the image display section 20. Therefore,it is possible to accurately detect a position pointed by the user withthe pointer. It is possible to display an image in a positioncorresponding to the position where the user visually recognizes thetarget.

Note that a method of the position detecting section 162 recognizing thepointer and an object (including a target) in the real space other thanthe pointer is not limited to the method of recognizing an image of thetarget on the basis of feature values of the image as explained above.For example, the target and the pointer may be selected from an objector the like included in an outside scene image according to aninstruction of the user. In this case, an instruction of the user may bean instruction by voice. The sound processing section 187 convertscollected by the microphone 63 into a text, whereby the positiondetecting section 162 acquires information for recognizing andspecifying the target. For example, when voice for designating featuresof the target in a picked-up image such as a color and a shape of thetarget is converted into a text, the position detecting section 162detects an image corresponding to the designated feature from thepicked-up image and recognizes the picked-up image.

A method of inputting information concerning the target may be operationon the track pad 14. The control section 140 may detect gestureoperation by a finger or the like. In this case, the user only has tomove a finger or the like and perform gesture operation in an imagepickup range of the right camera 61 and the left camera 62. The controlsection 140 only has to detect the gesture operation. Specifically, amethod may be adopted in which the user points the target to designatethe target itself. The user may use a gesture for, for example, pointingthe direction of the target with a hand or a finger or surrounding, witha hand or a finger, a range in which the user visually recognizes thetarget.

The AR-display control section 163 reads out content data (not shown inthe figure) stored by the storing section 120, controls the imageprocessing section 160 and the display control section 190, and causesthe image display section 20 to display an image for the AR display.When the content data includes sound data, the AR-display controlsection 163 controls the sound processing section 187 to output sound ofcontents from the right earphone 32 and the left earphone 34.

The AR-display control section 163 controls a display position of the ARcontents on the basis of relative positions of the target detected bythe position detecting section 162 and the display region and relativepositions of the pointer and the display region. That is, the AR-displaycontrol section 163 performs the AR display for displaying an image,characters, or the like in a position corresponding to the target. Thedisplay position of the AR contents may be either a position overlappingthe position or a position around the target. Consequently, the HMD 100provides information concerning the target or changes appearance of afigure of the target seen through the image display section 20. The ARcontents include data of the image or the characters displayed in theposition corresponding to the target. The AR-display control section 163determines the display position of the AR contents corresponding to thedetected target and displays the AR contents.

The AR-display control section 163 may control an angle of convergenceof the AR display according to a distance in which the user visuallyrecognizes the target detected by the position detecting section 162.The angle of convergence is an angle formed by the visual lines RD andLD shown in FIG. 3B. As the distance from the image display section 20(or the eyes RE and LE of the user) to the target OB is longer, theangle is smaller. When the user visually recognizes the AR imagedisplayed by the image display section 20, the angle of the visual linesRD and LD is determined according to the display position of the ARimage in the display region of the image display section 20. A distancein which the user senses the AR image changes according to the angle.For example, when display positions of the AR image on the left andright are set to increase the angle formed by the visual lines RD andLD, for the user, the AR image is seen as if the AR image is present ina position close to the user. The AR-image control section 163 mayadjust the display position of the AR image in the display region of theimage display section 20 according to the distance to the AR image thatthe AR-display control section 163 desires the user to sense. Processingrelated to the adjustment can be referred to as angle of convergencecontrol.

As explained in detail below, when the user performs work (a motion) ona target detected by the position detecting section 162, the AR-imagedisplay section 163 performs the AR display of a work item such that thework item can be visually recognized together with the target. In thiscase, the AR-image display section 163 adjusts the display position ofthe AR image in the display region of the image display section 20 suchthat the distance in which the user senses the AR image matches thedistance to the target detected by the position detecting section 162.Consequently, the user can naturally view the target and the AR image asif the user views objects present as the same distance. It is possibleto reduce a burden on the user.

In this embodiment, the AR-display control section 163 displays data ofitems included in the scenario data 123 on the basis of the scenariodata 123 stored by the storing section 120. That is, an image in whichthe scenario data 123 is displayed in a predetermined display formexplained below is equivalent to the AR contents.

FIGS. 5A and 5B are schematic diagrams showing configuration examples ofdata stored by the storing section 120. FIG. 5A shows a configurationexample of the scenario data 123 and the history data 124. FIG. 5B showsanother configuration example of the history data 124.

The scenario data 123 includes items. Order is associated with theitems. The order is relative order among the items. When the scenariodata 123 includes one item, order does not have to be associated withthe item.

The respective items included in the scenario data 123 correspond tomotions. The motions indicate behaviors or acts performed by the user.The motions may be movements of the body of the user or may includethoughts and determinations. One item of the scenario data 123 maycorrespond to one motion. One item of the scenario data 123 maycorrespond to a sequence including a plurality of motions.

The scenario data 123 illustrated in FIG. 5A is data of a scenarioconcerning a procedure of work and includes a plurality of items 123 arespectively corresponding to works. The respective items 123 a arenamed “work 1”, “work 2”, “work 3”, and the like. Contents (workcontents) of the items are associated with the respective items 123 aand included in the scenario data 123.

The scenario data 123 shown in FIG. SA is a form in which the historydata 124 is integrated. The history data 124 is data concerning ahistory of execution of the items included in the scenario data 123.When the scenario data 123 is a scenario of the works, the history data124 is data concerning a progress history and an execution history ofthe works corresponding to the items of the scenario data 123. In theexample shown in FIG. 5A, the history data 124 includes executionhistory data 124 a and execution detail data 124 b. The executionhistory data 124 a is data stored in association with the respectiveitems 123 a and indicates execution states (which can also be consideredexecution situations, progress situations, or progress statuses) of theworks corresponding to the items 123 a. In the example shown in FIG. 5A,when the works corresponding to the items 123 a have been completed,“completed” is set as the execution history data 124 a. Besides, theexecution history data 124 a such as executed and unexecuted can be set.

The execution detail data 124 b includes more detailed data concerning ahistory of execution of the works corresponding to the items 123 a. Forexample, the execution detail data 124 b includes data such as date andtime of execution of the works of the respective items 123 a included inthe scenario data 123, a progress degree of the works, and date and timeof completion of the works. The execution detail data 124 b may includepicked-up image data picked up by the right camera 61 and/or the leftcamera 62 when the works corresponding to the items 123 a are executed.The picked-up image data may be still image data or moving image data ormay include data of image pickup date and time.

The scenario data 123 may include image data 123 b in association withthe items 123 a. The image data 123 b may be still image data or may bemoving image data.

In FIG. 5B, another configuration example of the scenario data 123 andthe history data 124 is shown. FIG. 5B shows, in particular, an examplein which execution order of the items 123 a and a history of workscorresponding to the items 123 a are displayed as a time chart. Forexample, the time chart shown in FIG. 5B is displayed on the displaysection 20 according to control by the AR-display control section 163.The time chart includes execution order of works 1, 2, and 3corresponding to the items 123 a and an indicator (a mark) of an arrowshape indicating the present work situation. For example, when theAR-display control section 163 informs work based on the scenario data123 and the user executes the work, the AR-display control section 163can display the time chart shown in FIG. 5B and inform the user of anexecution state of work.

When the HMD 100 informs the user of work or the like on the basis ofthe scenario data 123, an execution state and the like of the work canbe input according to input operation by the user.

An input method in the HMD 100 is explained.

FIGS. 6A and 6B are explanatory diagrams of the input method in the HMD100, FIG. 6A shows an input example in which a virtual keyboard is used.FIG. 6B shows an example of a position input.

In FIGS. 6A and 6B, VR indicates a visual field of the user and V1indicates a region where the HMD 100 can display an image and cause theuser to visually recognize the image, that is, the display region of theimage display section 20.

For example, the display region V1 is located substantially in thecenter of the visual field VR of the user and is narrow than the visualfield VR. The display region V1 may have size same as the size of thevisual field VR. The size and the position of the display region V1 arenot limited to the examples shown in FIGS. 6A and 6B.

In the input method shown in FIG. 6A, the user performs an input using ahand of the user in a state in which a virtual keyboard V11 is displayedin the display region V1. A display position of the virtual keyboard V11can be a position set in advance. The AR-display control section 163displays the virtual keyboard V11 on the basis of image data (not shownin the figure) or the like of the virtual keyboard V11 stored in thestoring section 120. The position detecting section 162 detects a righthand HR and a left hand HL of the user in the visual field VR andcalculates positions of fingertips. Regions where the position detectingsection 162 is capable of detecting a position are image pickup rangesof the right camera 61 and the left camera 62. The regions may beoutside the display region V1. When the positions of the fingertipscalculated by the position detecting section 162 and display positionsof keys of the virtual keyboard V11 overlap, the AR-display controlsection 163 detects operation on the keys of the virtual keyboard V11.

FIG. 6B shows an input method for performing position pointing operation(position input operation) with a hand of the user in a state in whichthe virtual keyboard V11 is not displayed in the display region V1. Theinput method has an advantage that the input method can be executed evenif nothing is displayed in the display region V1 and, for example, doesnot affect the visibility of an outside scene seen through the imagedisplay section 20. The position detecting section 162 detectsfingertips of the hand (in the figure, the right hand HR) of the userfrom picked-up image data and calculates a position of a fingertip P.The AR-display control section 163 acquires, as an input position, theposition of the fingertip P calculated by the position detecting section162. The position detecting section 162 may be configured to, forexample, output a coordinate indicating a relative position of thefingertip P with respect to the display region V1. The positiondetecting section 162 may be configured to output a coordinate of arelative position of the fingertip P with respect to the visual fieldVR.

FIGS. 7A, 7B, and 7C are diagrams showing display examples in the HMD100 and show examples of display forms in which the items of thescenario data 123 are displayed according to the control by theAR-display control section 163.

In the examples shown in FIGS. 7A, 7B, and 7C, the target OB is includedin the visual field VR of the user. The user visually recognizes thetarget OB through the image display section 20. The position detectingsection 162 detects the position of the target OB, which is a target onwhich the AR display is performed, from a picked-up image of the camera61.

In the example shown in FIG. 7A, the AR-display control section 163displays a list SR in which the items of the scenario data 123 aredisplayed as a list. The list SR is a list in which the names of theitems included in the scenario data 123 are arranged according to theorder of the items set in the scenario data 123. In the list SR,contents of the items may be displayed together with the names (titles)of the items.

In the list SR, checkboxes CH are arranged in association with displaypositions of the items. The checkboxes CH indicate that motionscorresponding to the items have been completed. When the state detectingsection 164 detects completion of a motion, the AR-display controlsection 163 changes a display state of the checkbox CH to a stateindicating the motion completion. In this case, according to the controlby the AR-display control section 163, concerning an item for which themotion has been completed and the checkbox CH has been checked, adisplay color may be changed or a display size may be reduced toindicate that the motion has been completed. The AR-display controlsection 163 may change the display color to enable the user todistinguish, from the other items, an item of a motion to be executedfollowing the motion of the checked item and visually recognize theitem. When the checkbox CH is checked concerning the item, theAR-display control section 163 may set, in the scenario data 123, a timestamp indicating date and time when the motion is completed.

When the scenario data 123 includes one item, the list SR includes aname and the checkbox CH concerning the one item.

The list SR may not include the checkboxes CH. In this case, theAR-display control section 163 may delete, from the list SR, an item forwhich completion of a motion is detected by the state detecting section164. The AR-display control section 163 may change a display color ordisplay luminance in the list SR. For example, concerning the item forwhich the motion has been completed, the AR-display control section 163may change the display color or may reduce the display size to indicatethat the motion has been completed. The AR-display control section 163may change the display color to enable the user to distinguish, from theother items, an item of a motion to be executed following the completedmotion and visually recognize the item.

As shown in FIG. 7B, in the list SR, a marker M1 may be displayed tooverlap an item for which a motion has been completed. Further,concerning an item for which a motion is not completed or an item forwhich a motion is not performed and the next motion is started, a markerM2 indicating motion incompletion may be displayed.

An example of FIG. 7C is an example in which one of the items of thescenario data 123 is displayed or the items are displayed by a numberset in advance at a time. In this example, in the display region V1, onename or one name and one content of an item of the scenario data 123 aredisplayed in a display section D of a balloon form. An end portion ofthe display section D may be formed in an arrow shape to indicate aposition set as a target of a motion in the target OB. A marker M3 maybe displayed in a target position of a motion corresponding to the itemdisplayed in the display section. D. In the example shown in FIG. 7C,when the state detecting section 164 detects that a motion correspondingto the item displayed in the display section D has been completed, theAR-display control section 163 displays an item of the next orderincluded in the scenario data 123 on in display section D. In this way,in the display section D, the items included in the scenario data 123are sequentially displayed according to execution order of the motionsof the items.

Display positions and display forms of the list SR and the displaysection D can be changed as appropriate. For example, the list SR andthe display section D may be displayed in a position overlapping thetarget OB of a motion corresponding to an item of the scenario data 123.If the list SR and the display section D are displayed in a position notoverlapping the target OB, there is an advantage that the list SR andthe display section D do not hinder the motion and the user can easilymove while looking at the item. The list SR and the display section Dmay be displayed three-dimensionally (stereoscopically) or may bedisplayed as a two-dimensional (plane) image. Further, the list SR andthe display section D may be displayed in only one of the right displaydriving section 22 and the left display driving section 24. TheAR-display control section 163 may display the list SR and the displaysection D according to display order or importance of a motion or maymore clearly display a motion set in the scenario data 123 as a motionthat is particularly easily mistaken. Specifically, concerning thedisplay of the list SR and the display section D, the AR-display controlsection 163 may perform highlighted display, enlarged display, a changein a display color, or the like according to display order or importanceof a motion. The AR-display control section 163 may perform thehighlighted display, the enlarged display, the change in the displaycolor, or the like concerning a motion that is particularly easilymistaken. The AR-display control section 163 may shift, not to spoil thevisibility of a target of a motion visually recognized through the imagedisplay section 20, display positions of the list SR and the displaysection D from the center of the target. Further, concerning an item setin the scenario data 123 as particularly fine work, the AR-displaycontrol section 163 may enlarge and display an item and an image in thelist SR and the display section D. Further, concerning an item set inthe scenario data 123 as complicated work, the AR-display controlsection 163 may display a moving image in the list SR and the displaysection D according to, for example, an input of the user.

When the list SR and the display section D are displayed, the time chartillustrated in FIG. 5B may be displayed. In this case, concerning theplurality of items, display with which the user can easily graspexecution order of motions can be performed. When position pointingoperation for designating a position of the time chart is performed, anitem corresponding to the position pointed in the time chart may bedisplayed. Concerning the item corresponding to the pointed position,image data included in the scenario data 123 may be displayed.

When an item is displayed in the list SR and the display section D, animage based on the image data 123 b included in the scenario data 123illustrated in FIG. 5A may be displayed together with the item. The listSR and the display section D and the image may be switched anddisplayed.

With the display of the list SR and the display section D, it ispossible to meet needs of the user to, for example, check up to whichmotion in a series of motions including a plurality of motions (works)the user has completed work and create a record for securingtraceability of work.

The state detecting section 164 (FIG. 4) detects an execution state of amotion (work) corresponding to the item 123 a of the scenario data 123while the AR-display control section 163 performs the display based onthe scenario data 123.

The state detecting section 164 performs processing for detecting anexecution state by, for example, analyzing picked-up image data of theright camera 61 and/or the left camera 62. In this case, the statedetecting section 164 detects, from the picked-up image data, at leastany one of a target of work, an instrument or a tool used in the work,the body of the user, and the like. The state detecting section 164generates data concerning an execution state of the work on the basis ofpositions, sizes, colors, shapes, and the like of the target of thework, the instrument or the tool used in the work, the body of the user,and the like. The data concerning the execution state are, for example,unexecuted, under execution, execution completed, and the like.

In a state in which the AR-display control section 163 displays an itemof the scenario data 123, the state detecting section 164 may detect anexecution state of a motion corresponding to the item on the basis ofoperation by the user. In this case, operation detected by the statedetecting section 164 may be operation on the operation section such asthe track pad 14 of the control device 10. The state detecting section164 may receive an input by the input method explained with reference toFIGS. 6A and 6B.

For example, in an example shown in FIG. 7A, when the user performsoperation for designating the position overlapping the checkbox CH withthe pointer as shown in FIG. 7B, the state detecting section 164 maydetect that the motion has been completed. When the user performs agesture set in advance with the pointer, the state detecting section 164may detect that the motion has been completed. By analyzing voicecollected by the microphone 63 with the sound processing section 187,the user may performs an instruction input with voice to indicate thatthe motion has been completed. The state detecting section 164 maydetect on the basis of the instruction input that the motion has beencompleted. The state detecting section 164 may detect the completion ofthe motion on the basis of operation on the direction key 16, the trackpad 14, or the like included in the control device 10.

After the AR-display control section 163 starts the display based on thescenario data 123, the state detecting section 164 may acquire picked-upimage data of the right camera 61 and the left camera 62 and store theacquired picked-up image data as data concerning an execution state of amotion. In this case, for example, like a drive recorder, the statedetecting section 164 may cause the storing section 120 to storepicked-up image data during the motion.

The state detecting section 164 may generate a time chart (e.g., FIG.5B) that reflects the detected execution state of the motion. The timechart may be a comparison time chart to which a comparison resultrepresenting a difference from a standard time set in advance is given.

The communication control section 165 controls the communication section117 and the near-field-radio communication section 119 to execute radiocommunication with an external apparatus, an IC tag, or the like notshown in the figure and acquires data.

FIGS. 8A and 8B are schematic diagrams showing another configurationexample of the scenario data 123 stored by the storing section 120. FIG.8A shows a configuration example of scenario data 123 e. FIG. 8B shows aconfiguration example of display setting data 126.

The scenario data 123 e shown in FIG. 8A is used instead of the scenariodata 123 (FIG. 5A). Like the scenario data 123, the scenario data 123 eincludes items. Order is associated with the items. The respective itemsincluded in the scenario data 123 e correspond to motions. The scenariodata 123 e includes a plurality of items corresponding to procedures ofworks. The respective items are named “work 1”, “work 2”, “work 3”, andthe like. In the example shown in FIG. 8A, data of work procedures infive stages of the work 1 to the work 5 are included. The respectiveitems are associated with contents (work contents) of the items.

In this example, items concerning work for performing an overhaul of abearing are included. For example, “bearing cleaning” is associated withthe work 1 as work content. “Oil seal removal” is associated with thework 2. Targets are set for the items. The target of the work 1 is“bearing”. The target of the work 2 is “oil seal”.

The setting data 121 includes data of feature values for detecting thetargets stored in the scenario data 123 e from picked-up image data ofthe right camera 61 and the left camera 62.

Like the scenario data 123, flags indicating whether the motions of theitems have been completed can be set to correspond to the items. Whenthe items corresponding to the items are set, the HMD 100 can use thescenario data 123 e as history data.

FIG. 8B shows a configuration example of the display setting data 126for performing display based on the scenario data 123 e. The displaysetting data 126 is data defining a display form in performing displayin the visual field VR of the user according to the scenario data 123 e.The display setting data 126 is stored in, for example, the storingsection 120. For the targets corresponding to the items included in thescenario data 123 e, the display setting data 126 sets a display from indisplaying information to allow the user to easily visually recognizethe targets. In the example shown in FIG. 8B, when colors are displayedto be superimposed on the items corresponding to the items included inthe scenario data 123 e, the display setting data 126 associates thecolors and a progress situation of work.

The display setting data 126 sets color distinction corresponding to theorder of the works. As information for designating the colors, thedisplay setting data 126 may include information for designatingspecific colors, may include information for designating transmittancesof the colors, or may include information for designating luminance.When the colors are grouped into dark colors and light colors, thedisplay setting data 126 may include information for designating groups.The display setting data 126 may include information for designatinghighlighted display such as a fluorescent color.

In the example shown in FIG. 8B, a light color with low luminance is setto be displayed to be superimposed on a target corresponding to an itembefore work. The low luminance and the light color have an advantagethat the target on which the work is about to be performed is easilyseen. A light color is set to be displayed to be superimposed on atarget corresponding to an item during work. The light color has anadvantage that the light color does not hinder the work under execution.A dark color with high luminance is set to be displayed to besuperimposed on a target corresponding to particularly important work.For example, in the scenario data 123 e, if flags indicating importantworks can be set in association with the items, the targets of theimportant works can be distinguished from the other works usingdifferent colors and displayed. A dark color with low luminance is setto be displayed to be superimposed on a target corresponding to an itemfor which work has been completed. For the target for which the work hasbeen completed, since the end of the work only has to be visuallyrecognized, it is advantageous to set a dark color with low luminancesuch that the target can be clearly intuitively distinguished by visionfrom the other targets.

If the display setting data 126 is used, when the user performs workaccording the scenario data 123 e, it is possible to perform coloreddisplay for the targets in a state in which the targets aredistinguished using colors according to work order. In this case, theuser can distinguish, with the colors corresponding to the order of theworks, the targets for which the works should be performed and recognizethe target. Therefore, even a user unaccustomed to work can perform thework without mistaking a procedure.

A display example in which the scenario data 123 e and the displaysetting data 126 are used is explained. In the following explanation, aplurality of colors can be displayed by the image display section 20.The colors are indicated by hatching in the figure. Note that,naturally, it is also possible to display hatching of predeterminedcolors to be superimposed on the targets with the image display section20.

FIGS. 9A, 9B, 9C, and 9D are diagrams showing display examples in theHMD 100 and show the display examples in which AR display is performedusing the scenario data 123 e and the display setting data 126. FIGS.9A, 9B, 9C, and 9D show display examples in which an overhaul of abearing 300 is performed. The bearing 300, which is a target of work isvisually recognized in the visual field VR. Reference numeral 301denotes an outer ring of the bearing 300, 302 denotes an inner ring ofthe bearing 300, 303 denotes halls, and 304 denotes an oil seal.

A state before the work is shown in FIG. 9A. The bearing 300 is visuallyrecognized in the visual field. VR of the user. The AR-display controlsection 163 detects the bearing 300, the balls 303 of the bearing 300,and the oil seal 304 as targets on the basis of the scenario data 123 eand the setting data 121. Further, the AR-display control section 163specifies the positions of the bearing 300, the balls 303, and the oilseal 304 in the visual field VR and specifies correspondence between thepositions and the display region of the image display section 20.

FIG. 9B shows a state in which a predetermined color is displayed to besuperimposed on the oil seal 304. A marker M11 of the color is displayedin order of the scenario data 123 e and a form set in the displaysetting data 126 to overlap the oil seal 304. The AR-display controlsection 163 forms the marker M11 having a frame corresponding to theshape of the target of the AR display. The marker M11 is displayed to besuperimposed on the target in a state in which the marker M11 is paintedout by a color set in the display setting data 126. Shapes and sizes ofthe marker M11 and a marker M12 explained below may be determined by thedisplay setting data 126 and the scenario data 123 e. For example, whena target of work to be executed next is the oil seal 304, as shown inFIG. 93, the marker M11 is displayed to be superimposed on the oil seal304.

FIG. 9C shows a state after work for detaching the oil seal 304 has beencompleted. The oil seal 304 and the bearing 300 from which the oil seal304 is detached are visually recognized.

The AR-display control section 163 performs the AR display of the markerM12 in a position overlapping the balls 303 in order to inform work forremoving grease around the balls 303 performed following the work fordetaching the oil seal 304. This state is shown in FIG. 9D.

In FIG. 9D, the marker M12 having a ring-like frame is displayed tooverlap the balls 303. The marker M12 is painted out in a colordesignated in the display setting data 126. It is possible toappropriately adjust the visibility of the balls 303 by adjusting thetransmittance and the luminance of the color of the marker M12.

In this way, the distinguishing display by colors by AR is performed onthe target of the work, which is the object present in the real space,according to the procedure of the work and whether the work has beencompleted. Consequently, the user can easily distinguish, according to acolor, whether the work has been completed or not completed.

FIGS. 10A, 10B, 10C, and 10D are diagrams showing display examples inthe HMD 100 and show the display examples in which the AR display isperformed using the scenario data 123 e and the display setting data126. FIGS. 10A, 10B, 10C, and 10D respectively show examples in whichtexts and the like for informing work in addition to the distinguishingdisplay by colors in the examples shown in FIGS. 9A, 9B, 9C, and 9D.

FIG. 10A shows a state before the work 1 of the scenario data 123 e(FIG. 8A) is performed. The AR-display control section 163 displays,according to the scenario data 123 e, an explanation box TX1corresponding to the state before the work 1 is performed and includinga text for informing content of the work 1. The AR-display controlsection 163 may display a text indicator TX11 for informing that thebearing 300 is a bearing.

After an input indicating that the work 1 has been completed, theAR-display control section 163 displays an explanation box TX2corresponding to the work 2 as shown in FIG. 10B. In FIG. 10B, a textindicator TX12 corresponding to the oil seal 304, which is the target ofthe work 2, is displayed. As explained with reference to FIG. 9B, the ARdisplay of the marker M11 is performed to display the marker M11 to besuperimposed on the oil seal 304.

FIG. 10C shows a state after the work for detaching the oil seal 304 hasbeen completed. The oil seal 304 and the bearing 300 from which the oilseal 304 is detached are visually recognized.

The AR-display control section 163 performs display of a marker and theAR display of an explanation box corresponding to the work for removinggrease around the balls 303. This state is shown in FIG. 10D.

In FIG. 10D, the marker M12 is displayed to be superimposed on the balls303. An explanation box TX3 for explaining and informing content of thework 3 is displayed. A name of the balls 303 is displayed by a textindicator TX13 with respect to the balls 303, which are the target ofthe work 3.

For example, contents of texts displayed as the explanation boxes TX1 toTX3 and the text indicators TX11 to TX13 may be included in the scenariodata 123 e or may be stored in the storing section 120 in associationwith the items of the scenario data 123 e. Content of the AR displayperformed together with the display of the marker by the AR-displaycontrol section 163 is not limited to the text and may be a still imageor a moving image. A display color and a display size of the content ofthe AR display performed together with the display of the marker by theAR-display control section 163 may be set by the AR-display controlsection 163. Data designating the display color and the display size maybe included in the scenario data 123 e or the like and stored. Forexample, a display color of the texts displayed as the explanation boxesTX1 to TX3 and the text indicators TX11 to TX13 may be adjusted tocontent set as a display form of the marker in the display setting data126.

In the display examples shown in FIGS. 9A to 10D, the markers M11 to M12are explained as being distinguished and displayed using the colors setin the display setting data 126. However, the display forms set in thedisplay setting data 126 are not limited to the colors. For example,shapes of hatching may be set. Presence or absence and forms ofhighlighted display such as flashing and thick-bordered box display maybe set.

FIG. 11 is a diagram showing a display example in the HMD 100 and is adiagram showing another example of the AR display based on the scenariodata 123 e.

A bicycle 320, which is a target of work, is visually recognized in thevisual field VR. In the bicycle 320, a saddle 321, a hub bearing 322,and a tire 323 are set as targets of works by the scenario data 123 e.

In the display example shown in FIG. 11, an explanation box TX4including texts and the like for informing procedures of a plurality ofworks is displayed in the visual field VR. Like the list SR shown inFIG. 7A, the explanation box TX4 can also be considered a list of worksto be executed. The explanation box TX4 is not limited to the list. Forexample, the explanation box TX4 may include texts and the like forexplaining in detail contents of the respective works to be executed.

In the explanation box TX4, a plurality of works and order of the worksare displayed in association with each other. In the explanation boxTX4, color display sections TX31 to TX33 indicating display colors ofmarkers corresponding to the respective works are arranged.

In the visual field VR, display regions A1, A2, and A3 are respectivelyarranged in the saddle 321, the hub bearing 322, and the tire 323, whichare the targets of the works in the bicycle 320. The AR-display controlsection 163 determines the positions of the display regions A1, A2, andA3 according to the positions of the targets. The sizes of the displayregions A1, A2, and A3 may be included in the scenario data 123 e, thedisplay setting data 126, or other data and stored in the storingsection 120. The AR-display control section 163 may set the sizes of thedisplay regions A1, A2, and A3 according to the sizes of the targets.

The display regions A1, A2, and A3 are respectively painted out indisplay colors corresponding to the works. The colors of the displayregions A1, A2, and A3 are set in advance for each of the procedures ofthe works. For example, the colors of the display regions A1, A2, and A3may be set according to the execution order of the works and the settingof the display setting data 126 or may be designated in advance in thescenario data 123 e. The colors of the display regions A1, A2, and A3may be hatching or may be highlighted display like the markers M11 toM12 explained above.

In FIG. 11, in the explanation box. TX4, a color of the target of thework 1 is displayed in the color display section TX31 corresponding tothe work 1. The display region A1 is arranged to be superimposed on thesaddle 321, which is the target of the work 1. The display region A1 ispainted out in a color same as the color of the color display sectionTX31. The display region A2 is arranged to be superimposed on the hubbearing 322, which is the target of the work 2. The display region A2 ispainted out in a color same as the color of the color display sectionTX32. The display region. A3 is arranged to be superimposed on the tire323, which is the target of the work 3. The display region A3 is paintedout in a color same as the color of the color display section TX33.

In this way, by the AR-display control section 163, in the visual fieldVR, the regions corresponding to the targets of the works and theregions are distinguished using the colors and displayed according tothe order of the works and whether the works have been completed.Consequently, it is possible to clearly inform the user of the order ofthe works and the work targets.

By highlighting and displaying a detailed procedure (scenario guidance)of actual work with the explanation box TX4, it is possible to providedetailed information concerning a work procedure. The explanation boxTX4 may include, for example, concerning the contents of the works 1 to3, information for informing a method of detaching a cover of a bearing,a method of removing grease and bearing balls, a cleaning method, andthe like. More detailed work equivalent to know how and not included inordered items such as check of scratches on a bearing surface anddesignation of a type of new grease to be applied may be ordered anddisplayed. In this case, there is an advantage that it is possible toprovide detailed information called know how and technical informationcalled skill. For example, the AR-display control section 163 mayprovide information equivalent to the know how and the skill with popupdisplay or voice according to operation by the user. In this case,detailed information or technical information required by the userconcerning work can be provided in a timely manner.

FIGS. 12A and 12B are diagrams showing display examples in the HMD 100,diagrams showing other examples of the AR display based on the scenariodata 123 e, and modifications of the display shown in FIG. 11. In theexamples shown in FIGS. 12A and 12B, the display of the display regionsA1 to A3 is changed according to the progress of works.

In the example shown in FIG. 12A, as in FIG. 11, the explanation box TX4is displayed. In FIG. 12A, the display region A1 overlapping the saddle321, which is a target of work to be executed next, is displayed incolor. The other display regions A2 and A3 are not distinguished anddisplayed using colors.

According to an input indicating that the work targeting the saddle 321has been completed, as shown in FIG. 12B, the AR-display control section163 stops the color display of the display region A1 and displays, incolor, the display region A2 overlapping the hub bearing 322, which is atarget of work to be executed next.

As shown in FIGS. 12A and 12B, by changing the display of the displayregions A1 to A3 according to the progress of the works, a target ofwork to be executed next is clearly highlighted and displayed.Therefore, according to order of the works, it is possible to highlightthe target and cause the user to sense the target.

In this way, by displaying colors to be superimposed on objects orcomponents, which are targets of works, to distinguish the objects orthe components through the AR display having augmented reality, it ispossible to clearly inform the user of a specific target of work.Consequently, there is an effect that it is easy to associate languageinformation of a work standard and a real object. By changing colors ofthe display regions before and after execution of the workscorresponding to the items of the scenario data 123 e, it is possible touse display of the colors instead of a checklist.

The AR-display control section 163 may change, every time work iscompleted, display states of the items corresponding to the worksdisplayed in the explanation box TX4. For example, concerning completedwork, the AR-display control section 163 may change a display color ormay reduce a display size to indicate that the work has been completed.The AR-display control section 163 may change a display color such thatan item of work to be executed following the completed work can bedistinguished from the other items and visually recognized. When thework is completed and the display color is changed, the AR-displaycontrol section 163 may set, in the scenario data 123, a time stampindicating date and time of the completion of the work.

Concerning the display of the explanation box TX4, the AR-displaycontrol section 163 may more clearly display display order andimportance of motions or a motion set in the scenario data 123 as amotion particularly easily mistaken. Specifically, concerning thedisplay of the explanation box TX4, the AR-display control section 163may perform highlighted display, enlarged display, a change of a displaycolor, and the like according to the display order or the importance ofmotions. The AR-display control section 163 may perform the highlighteddisplay, the enlarged display, the change of the display color, and thelike concerning the motion particularly easily mistaken. The AR-displaycontrol section 163 may shift, not to spoil the visibility of a targetof a motion visually recognized through the image display section 20, adisplay position of the explanation box TX4 from the center of thetarget. Further, concerning an item set in the scenario data 123 asparticularly fine work, the AR-display control section 163 may enlargeand display the item and an image in the explanation box TX4. Further,concerning an item set in the scenario data 123 as complicated work, theAR-display control section 163 may display a moving image in theexplanation box TX4 according to, for example, an input of the user.

FIG. 13 is a flowchart for explaining the operation of the HMD 100.

When the user wears the HMD 100 and display of the scenario data 123 isinstructed by operation of the operation section 135, the controlsection 140 starts a motion related to the display (step S11).

The control section 140 acquires the scenario data 123 from the storingsection 120 (step S12) and selects a display form of the scenario data123 (step S12). In step S13, the control section 140 may select adisplay form set in advance or may select a display form designated byan input of the user. As the display form, there are, for example, theforms shown in FIGS. 7A, 7B, and 7C.

The control section 140 determines whether the history data 124 is used(reproduced) for the display of the scenario data 123 (step S14). Ifdetermining according to prior setting or operation by the user that thehistory data 124 is not reproduced (NO in step S14), the control section140 causes the image display section 20 to display the items of thescenario data 123 in the display form selected in step S13 (step S15).In step S15, the control section 140 controls display order of the itemsto match execution order of motions corresponding to the items of thescenario data 123.

Subsequently, the control section 140 detects a motion state on thebasis of picked-up image data of the right camera 61 and/or the leftcamera 62 and input operation by the user (step S16). The controlsection 140 creates a record of the detected motion state (step S17) anddetermines whether a motion corresponding to one item has been completed(step S18). If the motion has not been completed (NO in step S18), thecontrol section 140 returns to step S16.

If the motion has been completed (YES in step S18), the control section140 changes a display state of the item for which the motion isdetermined as having being completed among the items displayed on theimage display section 20 (step S19). The control section 140 stores thehistory data 124 in the storing section 120 or updates the history data124 stored in the storing section 120 on the basis of the record createdin step S17 (step S20). When the display form selected in step S13 is adisplay form for sequentially switching and displaying the items one byone or by a number set in advance and motions have been completed forall the displayed items, the control section 140 displays new items.

Thereafter, the control section 140 determines whether to complete thedisplay (step S21). If motions corresponding to all the items includedin the scenario data 123 have been completed or an end of the display isinstructed by operation by the user, the control section 140 determinesto end the display (YES in step S21) and ends the processing.

If determining not to end the display (NO in step S21), the controlsection 140 returns to step S16.

If determining in step S14 that the history data 124 is reproduced (YESin step S14), the control section 140 acquires the history data 124(step S22). The control section 140 displays the items included in thescenario data 123 starting from the next item of the item for which themotion is set as having been completed in the history data 124 (stepS23) and shifts to step S16.

The state detecting section 164 may detect an execution state on thebasis of an input recognized and detected by the position detectingsection 162. For example, the state detecting section 164 may detect theexecution state on the basis of voice, the position, the direction, orthe movement of a pointer, detection values of the nine-axis sensor 66,an image of a marker for input detected from picked-up images of theright camera 61 and/or the left camera 62, and operation of a footswitch(not shown in the figure).

As explained above, the HMD 100 according to the first embodimentapplied with the invention includes the image display section 20 thatdisplays an image to be visually recognizable through an outside sceneand the position detecting section 162 that detects an input in a rangevisually recognized through the image display section 20 and specifiesan input position. The control section 140 causes the image displaysection 20 to display an item on the basis of the scenario data 123including items and changes the display in the image display section 20according to the input position specified by the position detectingsection 162.

Consequently, a person performing a motion (e.g., work) targeting aplace or an object can view the item with the image display section 20and visually recognize the place or the object, which is the target ofthe motion, through the image display section 20. Therefore, the persondoes not need to perform a movement with a large load such as a largemovement of a visual line in order to view displayed content and caneasily view information.

The scenario data 123 includes a plurality of items corresponding tomotions. Order of items corresponding to execution order of the motionsis set in the respective items. Consequently, the user of the HMD 100can learn the execution order of the motions on the basis of a displayform of the items.

The AR-display control section 163 of the control section 140 mayarrange the plurality of items in a list format and cause the imagedisplay section 20 to display the items. In this case, the items arearranged according to the order set for the items. Consequently, theuser can learn the execution order of the motions from the arrangementof the displayed plurality of items.

The control section 140 may sequentially display the items in the orderset for the items. Consequently, the user can learn the execution orderof the motions from the display order of the items.

The scenario data 123 includes data including information concerning theexecution order of the motions corresponding to the items included inthe scenario data 123. The control section 140 causes the image displaysection 20 to display images such as the list SR and the display sectionD on the basis of the execution order. Consequently, it is possible tocause the user to visually recognize, with the images, the executionorder of the motions.

The AR-display control section 163 displays, in association with theitems, the checkboxes CH indicating completion of the motionscorresponding to the items. In this case, after causing the imagedisplay section 20 to sequentially display an item in the order set forthe item, when detecting completion of a motion corresponding to theitem, the AR-display control section 163 changes a display state of thecheckbox CH to a state indicating the completion of the motion.Consequently, it is possible to clearly show a progress state of themotion.

The HMD 100 stores the scenario data 123 and the order data in thestoring section 120 in association with each other. Therefore, it ispossible to easily read out and use a correspondence relation betweendata concerning the items concerning the motions and data concerning theexecution order of the motions and the respective data.

The control section 140 may display a marker functioning as an indicatorto be superimposed on an item displayed in a position corresponding tothe input position specified by the position detecting section 162. Inthis case, the user can change a display form of the displayed item byperforming an input.

The state detecting section 164 of the control section 140 may detect anexecution state of a motion corresponding to an item on the basis ofpicked-up images of an outside scene picked up by the right camera 61and the left camera 62 or voice collected from the outside scene by themicrophone 63. The AR-display control section 163 may change, accordingto the detected execution state, a display state of an item displayed bythe image display section 20. In this case, it is possible to change thedisplay state of the item according to the execution state of themotion.

The storing section 120 stores the history data 124 includinginformation related to execution states of motions. The control section140 causes the storing section 120 to store, in association with thescenario data 123, the history data 124 of motions corresponding to theitems included in the scenario data 123. When the history data 124 isread out from the storing section 120, the control section 140 controls,on the basis of the read-out history data 124, the display state of theitem displayed by the image display section 20. In this case, by storingdata concerning execution states of motions and reading out the data andreflecting the data on display, it is possible to record the executionstates and invoke the recorded execution states.

The control section 140 detects a target in an outside scene with theposition detecting section 162. The AR-display control section 163causes the image display section 20 to display the items included in thescenario data 123 to correspond to a position where the target detectedby the position detecting section 162 is visually recognized through theimage display section 20. Consequently, it is possible to easilyvisually recognize both of a place or an object of the target of themotion and an item.

In this way, the HMD 100 can perform the AR display of a check or thelike of work performed by a human to be superimposed on an object in areal world or an object of an electronic medium. In this case, byadjusting a form of the display to the position of a target, it ispossible to perform stabilizing display for retaining a relativeposition with respect to the target and retain an item in a positionwhere the item is easily visually recognized.

The control section 140 causes the image display section 20 to displayinformation related to a target selected from an object in a real spacevisually recognized through the image display section 20 in a displayform corresponding to the target. For example, as shown in FIGS. 9A to12B, the AR-display control section 163 performs the AR displaycorresponding to the bearing 300, the balls 303, the oil seal 304, thesaddle 321, the hub bearing 322, the tire 323, or the like set as atarget of work. The AR-display control section 163 performsdistinguishing display by colors functioning as information display onthe markers M11 and M12 and the display regions A1 to A3. Consequently,it is possible to show information concerning the object in the realspace visually recognized through the image display section 20 to theuser of the HMD 100 as if the information corresponds to the object inthe real world.

The control section 140 causes the image display section 20 to displayinformation for highlighting the target in the real space visuallyrecognized through the image display section 20 to be superimposed onthe target. For example, the AR-display control section 163 displays themarkers M11 and M12 to be superimposed on the balls 303 and the oil seal304 set as the targets of the works to highlight the balls 303 and theoil seal 304. For example, the AR-display control section 163 displaysthe display regions A1 to A3 to respectively superimpose the displayregions A1 to A3 on the saddle 321, the hub bearing 322, and the tire323 set as the targets of the works and highlights the saddle 321, thehub bearing 322, and the tire 323. Consequently, it is possible tohighlight an object in the real world related to work and show theobject to the user of the HMD 100.

The control section 140 switches, according to an input, informationdisplayed in a display form corresponding to a target to informationcorresponding to a different target and displays the information. Forexample, the AR-display control section 163 displays a color asinformation in the display region A1 displayed to be superimposed on thesaddle 321 set as the target of the work. The AR-display control section163 stops the display of the display region A1 according to an inputindicating that the work has been completed and displays a color in thedisplay region A2 displayed to be superimposed on the hub bearing 322.Therefore, it is possible to sequentially display different informationaccording to a different target.

In the embodiment, a method of changing, according to a progress stateof work (a motion), a display state of an item displayed by the imagedisplay section 20 is not limited to the method of changing a displaystate of the checkbox CH to a state indicating motion completion in thelist SR. For example, an item for which completion of a motion isdetected may be deleted from the list SR or a display color or displayluminance in the list SR may be changed. For example, a display color ofan item may be maintained and a color for improving visibility such as afluorescent color may be painted out in a position overlapping the item.When the display color of the item is changed, the transmittance of anoutside scene in the position overlapping the item may be changed byadjusting the display luminance. A display position of the item may bechanged. For example, the item for which the work has been completed maybe moved to a different row. Further, an item may be displayed on one ofthe left optical-image display section 28 and the right optical-imagedisplay section 26. A display position of the item for which the workhas been completed may be moved from the left optical-image displaysection 28 to the right optical-image display section 26 or in theopposite direction. Consequently, it is possible to adopt various“display change” forms as long as a procedure that has to be performedis clearly understand in display, although the procedure can be checkedafterward.

Application examples of the HMD 100 in the first embodiment includeexamples explained below.

When a standard operation manual of work including a plurality ofprocesses is formed by the scenario data 123, the scenario data 123 canbe used as a checklist. In this case, documents, still images, andmoving images for informing the work can be included in the scenariodata 123. Voice may be included in the moving images and included in thescenario data 123.

In the standard operation manual, components, apparatuses, or parts ofthe components or the apparatus set as work targets can be displayed foreach of processes of work as the target objects OB by the AR-displaycontrol section 163.

The scenario data 123 may be able to be edited on the basis of operationby the user during display based on the scenario data 123. As an inputmethod in this case, the method explained with reference to FIGS. 5A and5B can be adopted.

In this example, the user can wear the HMD 100, actually perform work,and reflect a change of the work, a changed point of a gist annotation,and the like on the scenario data 123 on a real time basis.

Moving images or still images picked up by the right camera 61 and theleft camera 62 and voice collected by the microphone 63 during the workmay be able to be included in the scenario data 123 and stored.

Examples of the work include line work in a factory and work in a cellproduction system. However, the invention can also be applied to supportof work in a shop or the like. By enabling the scenario data 123 to beedited, it is possible to register and store, as standard work, an imagethat could be the standard work while performing the work.

A process of inventory management of shelf merchandise in a small shopor the like can be performed according to the scenario data 123. In thiscase, the user wearing the HMD 100 can detect commodities with theposition detecting section 162 by making a tour of inspection in thestore and picking up images of the commodities with the right camera 61and the left camera 62. In this case, the number and the places ofcommodities, time, and the like can be recorded and stored as thehistory data 124. Consequently, the HMD 100 can also be used as a worksupporting system for an inventory management job, a picking job, andthe like.

In a diagnostic job in a hospital or a clinic, a diagnosis flow can becreated and utilized as the scenario data 123. For example, check itemsin an interview with a patient and items of a test result are includedas the items of the scenario data 123, whereby a diagnosis can beperformed according to the scenario data 123. In this case, data inputby medical personnel such as a doctor, who is a user, concerning theitems included in the scenario data 123 and picked-up image data of theright camera 61 and the left camera 62 only have to be stored and savedin the storing section 120 as the history data 124. In thisconfiguration, the history data 124 can be utilized as a part of anelectronic clinical record, that is, a diagnosis record.

The invention can also be applied when the user wearing the HMD 100counts the number of people or objects in a real space visuallyrecognized as an outside scene. In this case, it is effective that,every time input operation such as a gesture is performed, the countingadvances and the AR display in which an image or a marker indicatingthat the counting has been performed overlaps the counted people orobjects is performed. In this application example, it is possible toobtain effects that, for example, double count is prevented and a check(count) can be thoroughly performed, although the HMD 100 is operated ina hands-free manner. For example, the HMD 100 can be applied to, forexample, count of the number of people in an invent venue and count ofthe number of components during factory work.

The invention can also be applied when the user wearing the HMD 100 is areferee or a scorer of a ballgame. In this case, when a game isperformed in a real space that can be visually recognized as an outsidescene, count can be executed concerning plays during the game.Specifically, it is possible to prevent mistakes by performing count ofstrikes and balls count and display of an inning number, the first halfand the second half of an inning, scores, and the like in a game ofbaseball. For example, it is possible to use the HMD 100 for count ofthe number of rounds, calculation of scores, and the like in a golfcompetition and prevent mistakes. A player may wear the HMD 100.

In this way, by applying the invention using the HMD 100, it is possibleto guarantee accuracy of a motion of the user and realize a check of aprocedure, a check of the number of points, and a check, compensation,and the like for traceability for quality guarantee in a manufacturingsite. For example, it is possible to surely prevent, with the functionsof the HMD 100, mistakes, troubles, and the like due to uncertainty of abehavior, vagueness of memory, and the like of a human.

The scenario data 123 can be, for example, a presentation project filecreated in a form of presentation software (e.g., PowerPoint (registeredtrademark) of Microsoft (registered trademark) Corporation). In thiscase, the presentation project file is displayed as the scenario data123. The user wearing the HMD 100 can perform presentation while viewingdisplay in the display region V1. When sentences to be spoken in thepresentation are included in the scenario data 123 as contents of items,the scenario data 123 can be used as a manuscript of the presentation.In this case, voice uttered by the user is collected by the microphone63, converted into a text with the sound processing section 187, andcollated with the contents of the scenario data 123. Then, the controlsection 140 can specify a spoken text (characters or character strings)in the scenario data 123. If the AR-display control section 163 changesa display form of the spoken text, it is possible show a progress stateof the presentation to the user. For example, a form is conceivable inwhich the spoken text is erased from the display region V1. In thiscase, a display position and display luminance of an image or a text aredesirably set to a position and luminance that do not hinder thevisibility of an outside scene visually recognized through the imagedisplay section 20. Consequently, for example, the user can easilyvisually recognize slide images of the presentation displayed by aprojector. A text or the like to be spoken may be, for example,highlighted.

When an operator performs work while looking at, for example, a manualof a procedure of the work, a check concerning to which part the workhas been executed and whether the work has been executed in a correctprocedure depends on memory of the operator. Similarly, when a personexecutes presentation, it is not easy to record to which part thepresentation advances. The management of the recording depends on memoryof the person executing the presentation. If the HMD 100 in thisembodiment is used, the user wearing the HMD 100 can check work contentby himself or herself using a checklist. When a motion is detected onthe basis of picked-up image data, it is possible to easily and surelyperform a motion such as work corresponding to an item included in thescenario data 123 compared with when a check concerning whether themotion has been carried out, a check of a progress state, and the likedepends only on memory of the user. It is possible to prevent executionomission of a motion, a mistake of execution order of the motion, andthe like. Even if another motion is performed while the series ofmotions included in the scenario data 123 are sequentially performed, itis possible to obtain information concerning a suspended motion with thelist SR and the display section D displayed by the AR-display controlsection 163. Consequently, it is possible to improve accuracy of work,presentation, or the like including a plurality of processes and achieveimprovement in various aspects such as speed, accuracy, and managementof time.

Second Embodiment

FIG. 14 is a flowchart for explaining the operation of the HMD 100 in asecond embodiment.

The configuration of the HMD 100 in the second embodiment is common tothe configuration explained with reference to FIGS. 1 to 4 in the firstembodiment. Therefore, illustration and explanation of the configurationare omitted.

In the second embodiment, the HMD 100 acquires the scenario data 123with the near-field-radio communication section 119. The HMD 100 mayacquire the history data 124 together with the scenario data 123.

In the flowchart of FIG. 14, the same step numbers are added toprocessing common to the operation explained with reference to FIG. 13.

In the operation shown in FIG. 14, the near-field-radio communicationsection 119 stands by for communication according to control by thecommunication control section 165 (step S31). In step S31, thenear-field-radio communication section 119 attempts communication to anexternal apparatus. Specifically, the near-field-radio communicationsection 119 attempts, for example, transmission of a radio signal forperforming a search of the external apparatus and reception of a radiosignal transmitted from the external apparatus. The external apparatusis, for example, an apparatus, an IC tag, or the like that performscommunication through. Bluetooth or NFC. The communication controlsection 165 continues the attempt of the communication until thenear-field-radio communication section 119 becomes capable of startingthe radio communication with the external apparatus (NO in step S31). Ifthe communication by the near-field-radio communication section 119 hasbecome possible (YES in step S31), the communication control section 165causes the near-field-radio communication section 119 to execute theradio communication with the external apparatus and acquires thescenario data 123 (step S32). In step S32, the communication controlsection 165 may acquire the history data 124. The scenario data 123 orthe scenario data 123 and the history data. 124 acquired by thecommunication control section 165 are stored in the storing section 120.

The AR-display control section 163 reads out the scenario data 123acquired by the communication control section 165 and stored in thestoring section 120 and displays an item included in the scenario data123 (step S33). Concerning the display of the item, processing same assteps S14, S15, and S23 of FIG. 13 is performed. The control section 140executes the operation in steps S16 to S21.

The near-field-radio communication section 119 executes the near fieldradio communication as explained above. Therefore, when an externalapparatus is absent in a range in which the near-field-radiocommunication section 119 is capable of performing communication, thecommunication control section 165 does not acquire the scenario data123. When an external apparatus appears in the communicable range of thenear-field-radio communication section 119, the communication controlsection 165 acquires the scenario data 123 from the apparatus anddisplays the scenario data 123.

For example, when the control device 10 of the HMD 100 is brought closeto the apparatus that retains the scenario data 123, radio communicationoccurs in step S31 (YES in step S31). An item included in the scenariodata 123 acquired from the apparatus is displayed by the image displaysection 20.

When the HMD 100 in the second embodiment has approached the apparatusthat retains the scenario data 123, the HMD 100 performs display basedon the scenario data 123 retained by the apparatus that the HMD 100 hasapproached. In a place where a plurality of apparatuses that retain thescenario data 123 are present, the HMD 100 performs display based on thescenario data 123 retained by the apparatus that the HMD 100 hasapproached. In this case, the HMD 100 can acquire the scenario data 123corresponding to the place and perform display.

As a specific example, the apparatus communicable with the HMD 100 andhaving the scenario data 123 is disposed in a place of furniture such asa desk, utensils, or equipment in a building, a household electricappliance such as a refrigerator, a machine or a production lineconfigured by a plurality of machines in a factory, an automobile, orthe like. In this case, when the user wearing the HMD 100 approaches theapparatus having the scenario data 123, display based on the scenariodata 123 is performed. Therefore, it is possible to provide the userwith the scenario data 123 corresponding to a place where the user ispresent. If the scenario data 123 is configured to be modifiable byoperation by the user, the scenario data 123 can be modified at anytime. The modified scenario data 123 may be transmitted to the externalapparatus.

As explained above, the HMD 100 may include the communication section117 that communicates with an external apparatus and the ear-field-radiocommunication section 119 and receive the scenario data 123 from theexternal apparatus. In this case, it is possible to acquire dataconcerning items from the outside. The near-field-radio communicationsection 119 may execute the near field radio communication and acquirethe scenario data 123. In this case, since a distance in which the nearfield radio communication can be performed is limited, it is possible toacquire data associated with a place.

Modifications

A system including a plurality of the HMDs 100 explained in the firstand second embodiments can also be configured. In this case, as anexample, there is a form in which an administrator who administersmotions and an executor who executes motions respectively wear the HMDs100.

In this configuration, the HMD 100 of the administrator and the HMD 100of the executor can share the scenario data 123 and the history data 124by transmitting and receiving the scenario data 123 and the history data124 between the HMDs 100. That is, the HMD 100 of the executor transmitsthe scenario data 123 and the history data 124 to the HMD 100 of theadministrator. The HMD 100 of the administrator performs display basedon the scenario data 123 and the history data 124 received by the HMD100. Consequently, the administrator can learn an execution state of amotion of the executor.

A configuration may also be adopted in which the HMD 100 of theadministrator edits the scenario data 123 and transmits the scenariodata 123 after the editing to the HMD 100 of the executor and the HMD100 of the executor performs display based on the scenario data 123after the editing. In this case, the administrator can support theexecutor by editing the scenario data 123 to support execution of amotion of the executor. In this configuration, the administrator can beconsidered an information providing side, that is, an informationprovider and the executor can be considered an information receiver.

A plurality of different scenario data 123 may be associated with oneanother. For example, the scenario data 123 for the administrator andthe scenario data 123 for the executor can be associated with eachother. The scenario data 123 for the administrator and the scenario data123 for the executor may be different data. When an item checked by theadministrator and an item checked by the executor are different, thescenario data 123 for the administrator and the scenario data 123 forthe executor only have to include different items. The HMD 100 only hasto retain the history data 124 corresponding to the respective items.

In this case, concerning work executed by the executor, the HMDs 100respectively worn by the administrator and the executor can check anexecution state of a motion and generate the history data 124.Consequently, it is possible to perform a double check concerning theexecution of the motion. It is possible to achieve improvement ofexecution accuracy of the motion.

Note that the invention is not limited to the configurations of theembodiments and can be carried out in various forms without departingfrom the spirit of the invention.

In the embodiments, the configuration in which the user visuallyrecognizes the outside scene through the display section is not limitedto the configuration in which the right optical-image display section 26and the left optical-image display section 28 transmit the externallight. For example, the invention can also be applied to a displaydevice that displays an image in a state in which the outside scenecannot be visually recognized. Specifically, the invention can beapplied to a display device that displays, for example, picked-up imagesof the right camera 61 and/or the left camera 62, an image and a CGgenerated on the basis of the picked-up images, and a video based onvideo data stored in advance or video data input from the outside. Thedisplay device of this type includes a display device of a so-calledclosed type with which an outside scene cannot be visually recognized.Naturally, a display device that displays video data or an analog videosignal input from the outside without performing processing such as ARdisplay, MR display, or VR display is included as an application targetaccording to the invention.

For example, instead of the image display section 20, an image displaysection of another type such as an image display section worn like a capmay be adopted. The image display section only has to include a displaysection that displays an image corresponding to the left eye of the userand a display section that displays an image corresponding to the righteye of the user. The display device according to the invention may beconfigured as, for example, a head mounted display mounted on a vehiclesuch as an automobile or an airplane. For example, the display devicemay be configured as a head mounted display incorporated in a bodyprotector such as a helmet. In this case, a portion for positioning aposition with respect to the body of the user and a portion positionedwith respect to the portion can be set as a wearing section.

Further, in the embodiments, in the configuration explained as theexample, the image display section 20 and the control device 10 areseparated and connected via the connecting section 40. However, aconfiguration can also be adopted in which the control device 10 and theimage display section 20 are integrated and worn on the head of theuser.

As the control device 10, a notebook computer, a tablet computer, or adesktop computer may be used. As the control device 10, portableelectronic apparatuses including a game machine, a cellular phone, asmart phone, and a portable media player, other dedicated devices, andthe like may be used. A configuration may be adopted in which thecontrol device 10 and the image display section 20 are separated and thecontrol device 10 and the image display section 20 transmit and receivevarious signals each other through radio communication.

For example, as the component that generates image light in the imagedisplay section 20, the image display section 20 may include an organicEL (Electro-Luminescence) display and an organic EL control section. Asthe component that generates image light, an LCOS (Liquid Crystal onSilicon: LCoS is a registered trademark), a digital micro mirror device,or the like can also be used.

The “display section” in the invention is equivalent to a component thatemits image light. In the following explanation, the HMD 100 emittingthe image light is referred to as “display”.

In the embodiments, the left and right image-light generating sectionsexplained with reference to FIG. 4 generate image lights and the rightoptical-image display section 26 and the left optical-image displaysection 28 shown in FIG. 2 radiate the image lights respectively towardthe right eye and the left eye of the user. Consequently, the rightoptical-image display section 26 and the left optical-image displaysection 28 respectively make the image light incident on the right eyeand the left eye of the user. The configuration of the “display section”is not limited to this. That is, the configuration of the “displaysection” is not limited to the configurations shown in FIGS. 2 and 4 aslong as the display section radiates image light. For example, in theconfiguration in the embodiments, image lights are emitted toward theeyes of the user by the “right light guide section” and the “left lightguide section” including the half mirrors 261A and 262A. As componentsthat generate image lights, the HMD 100 includes the right backlight 221and the left backlight 222 and the right LCD 241 and the left LCD 242.The “display section” does not require these components.

For example, image lights generated by a mechanism incorporated in oneor both of the right display driving section 22 and the left displaydriving section 24 of the image display section 20 may be reflected by areflection mechanism provided on the user side of the image displaysection 20, that is, a side facing the eyes of the user and emitted tothe eyes of the user. As the reflection mechanism, a scanning opticalsystem including a MEMS (Micro Electro Mechanical Systems) mirror can beadopted. That is, the image display section 20 may include scanningoptical systems including MEMS mirrors that scan lights emitted by theimage-light generating sections. The lights scanned by the scanningoptical systems may be directly made incident on the eyes of the user.Optical members on which virtual images are formed by the lights scannedby the scanning optical systems may be provided in the image displaysection 20. The optical members form virtual images by the scanninglights scanned by the MEMS mirrors. In this case, the MEMS mirrors scanlights, whereby virtual images are formed on virtual image formingsurfaces. The user catches the virtual images with both the eyes tovisually recognize (recognize) image. Optical components in this casemay be optical components that guide lights through a plurality of timesof reflection like, for example, the right light guide plate 261 and theleft light guide plate 262 in the embodiments. Half mirror surfaces maybe used as the optical components.

The scanning optical systems are not limited to the configurationincluding the MEMS mirrors. The mechanisms that generate image lightsmay be laser beam sources that emit laser beams. For example, it is alsopossible to apply the invention to a head mounted display of a laserretinal projection type. That is, a configuration may be adopted inwhich light emitting sections include laser beam sources and opticalsystems for guiding laser beams to the eyes of the user, make the laserbeams incident on the eyes of the user to scan the retina, and formimages on the retina to thereby cause the user to visually recognize animage.

Instead of the virtual image forming surfaces that receive the scannedlights, components that guide the image lights to the eyes of the userusing diffraction gratings may be adopted. That is, the components arenot limited to components that guide the image lights inside the opticalmembers and may be components including only a function of refractingand/or reflecting the image lights toward the eyes of the user.

In the configuration including the scanning optical systems includingthe MEMSs or the like, it is possible changes a position where the uservisually recognizes an image, that is, a display position of the imageby changing an attachment angle of the scanning optical systems in theimage display section 20. Therefore, in the processing for changing thedisplay position in the embodiments, a motion for changing the angle ofthe scanning optical systems may be performed instead of the motion forchanging the display positions of the images in the right LCD 241 andthe left LCD 242.

As the optical systems that guide the image lights to the eyes of theuser, it is possible to adopt components including optical members thattransmit external light made incident toward the device from theoutside. The components make the external light incident on the eyes ofthe user together with the image lights. Optical members located infront of the eyes of the user and overlapping a part of or the entirevisual field of the user may be used.

In the embodiments, the configuration is illustrated in which virtualimages are formed by the half mirrors 261A and 262A on parts of theright optical-image display section 26 and the left optical-imagedisplay section 28 located in front of the eyes of the user. Theinvention is not limited to this. Images may be displayed in the displayregions having areas occupying the entire or most of the rightoptical-image display section 26 and the left optical-image displaysection 28. In this case, processing for reducing the image may beincluded in the motion for changing the display position of the image.

Further, the optical elements according to the invention are not limitedto the right light guide plate 261 and the left light guide plate 262including the half mirrors 261A and 262A and only have to be opticalcomponents that make image lights incident on the eyes of the user.Specifically, a diffraction grating, a prism, or a holography displaysection may be used.

At least a part of the functional blocks shown in FIG. 4 may be realizedby hardware or may be realized by cooperation of the hardware andsoftware. The invention is not limited to the configuration in which theindependent hardware resources are disposed as shown in FIG. 4. Thecomputer program executed by the control section 140 may be stored inthe storing section 120 or a storage device in the control device 10.Alternatively, the control section 140 may be configured to acquire thecomputer program stored in an external device via the communicationsection 117, the near-field-radio communication section 119, or theinterface 125 and execute the computer program. Among the componentsformed in the control device 10, only the operation section 135 may beformed as an independent user interface (UI). The components formed inthe control device 10 may be redundantly formed in the image displaysection 20. For example, the control section 140 shown in FIG. 4 may beformed in both of the control device 10 and the image display section20. The functions performed by the control section 140 formed in thecontrol device 10 and the CPU formed in the image display section 20 maybe separated.

The entire disclosure of Japanese Patent Application Nos. 2015-135738,filed Jul. 7, 2015 and 2016-013249, filed Jan. 27, 2016 are expresslyincorporated by reference herein.

What is claimed is:
 1. A head-mounted display device comprising: amemory configured to store scenario data associated with a targetobject, the scenario data including a work list image displaying anitem; a display configured to display the work list image; a cameraconfigured to capture an outside image; and a processor configured to:receive the outside image from the camera; detect the target object fromthe outside image; detect a pointer on the outside image; determinewhether a gesture is associated with the work list image based on aposition of the detected pointer, the gesture including a movement of afinger or a hand in a direction of the target object; detect completionof the gesture corresponding to the item; and control the display tochange a display state of the work list image to a state indicatingcompletion of the gesture corresponding to the item, wherein the displayis configured to display a first marker to overlap the item for whichthe gesture is completed, and wherein the display is configured todisplay a second marker indicating the gesture is not completed.
 2. Thehead-mounted display device according to claim 1, wherein the work listimage comprises a checkbox indicating the gesture corresponding to theitem is completed.
 3. The head-mounted display device according to claim1, wherein the processor is configured to detect the completion of thegesture corresponding to the item, when a user designates a positionoverlapping a part of the displayed work list image with the pointer. 4.The head-mounted display device according to claim 1, wherein theprocessor is configured to set, in the scenario data, a time stampindicating date and time when the gesture is completed.
 5. Thehead-mounted display device according to claim 1, wherein the display isconfigured to display a name or content associated with the item of thescenario data in a display section.
 6. The head-mounted display deviceaccording to claim 5, wherein the display section is formed in a balloonshape.
 7. The head-mounted display device according to claim 5, whereinen end portion of the display section is formed in an arrow shape toindicate a target position of the gesture.
 8. The head-mounted displaydevice according to claim 7, wherein the display is configured todisplay a third marker in the target position of the gesturecorresponding to the item displayed in the display section.
 9. Thehead-mounted display device according to claim 7, wherein the display isconfigured to display a plurality of regions that have different displaycolors, the plurality of the regions corresponding to targets of worksand being displayed according to an order of the works.
 10. Thehead-mounted display device according to claim 1, wherein the camera isconfigured to record an outside video and the processor is configured toreceive the outside video from the camera.
 11. The head-mounted displaydevice according to claim 1, wherein the display is a closed type withwhich a user does not visually recognize an outside scene.
 12. Thehead-mounted display device according to claim 1, wherein the outsideimage is an image of an outside scene in a front side direction of thehead-mounted display device.
 13. A method for a head-mounted displaydevice, the method comprising: storing scenario data associated with atarget object, the scenario data including a work list image displayingan item; displaying the work list image; receiving an outside image;detecting the target object from the outside image; detecting a pointeron the outside image; determining whether a gesture is associated withthe work list image based on the detected pointer, the gesture includinga movement of a finger or a hand in a direction of the target object;detecting completion of the gesture corresponding to the item; andchanging a display state of the work list image to a state indicatingcompletion of the gesture corresponding to the item, wherein thedisplaying includes displaying a first marker to overlap the item forwhich the gesture is completed, and wherein the displaying includesdisplaying a second marker indicating the gesture is not completed. 14.A non-transitory computer-readable medium comprising computer programinstructions that, when executed by a processor of a head-mounteddisplay device, cause the head-mounted display device to: store scenariodata associated with a target object, the scenario data including a worklist image displaying an item; display the work list image; receive anoutside image; detect the target object from the outside image;detecting a pointer on the outside image; determine whether a gesture isassociated with the work list image based on a position of the detectedpointer, the gesture including a movement of a finger or a hand in adirection of the target object; detect completion of the gesturecorresponding to the item; and change a display state of the work listimage to a state indicating completion of the gesture corresponding tothe item, wherein the displaying includes displaying a first marker tooverlap the item for which the gesture is completed, and wherein thedisplaying includes displaying a second marker indicating the gesture isnot completed.